![\"Tablet](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2026/05/tablet-viz.jpg\")
\nImaginary strings running across the tablet - shows a G chordPictured is Duke James playing a set at the Smalltown Society lab before the open mic. This was during\nthe Smalltown music lab on 5/6/2026. Duke played an amazing and soulful show
\nThis wasn't just my first open mic using Sketchatone, but my first open mic period.
\n\nA short clip captured of me at the open mic, see the bottom of the post for a home recording
\nI've been taking singing lessons for a few\nyears now, and been doing the "Band lab" thing with my instructor Jim Bedford.\nI've been working with a great set of musicians - and we're all learning, but doing great and playing a couple times a year as we form several versions\nof bands around the songs we're working on. I also did a Bandworks stint with some friends last year.
\nAll of it is fun, but inevitably in any of these setups there is the one guitar player who can do some decent\nrhythm playing and sing solo. I get jealous, try to do some stuff on keys while singing, and I'm just not vibing with it.
\nI can play keys decently in a band situation, but just not proficiently enough to fill the same space as a guitar - especially when singing.\nNot just that, but most of the music I grew up with is very guitar driven. I've tried jamming on keys\nto a guitar oriented song, but again its just not vibing most of the time. And when it does, it's a rather slow (and likely depressing) song.
\nNothing wrong with that, but that's not what I want to play right now.
\nAnyway, this was all really the motivation behind Sketchatone. I blogged about it back when I initially released it in February, but the gist is this:
\nGrab an inexpensive drawing tablet - and assuming its a model that can work with my software, that becomes your MIDI controlled guitar strummer.\nImagine some strings laid out horizontally across the tablet. Unlike a real guitar, we can have as many strings as you want.\nI'm typically working with 9 to span a 3 note chord across three octaves.
\n\nImaginary strings running across the tablet - shows a G chord
As you drag your pen across the tablet, it will hit and pluck each string. Drawing tablets can offer so much control in terms\nof pen tilt, pressure, etc. Sketchatone can map these to MIDI velocity, pitch bend, note sustain etc. It's\nall pretty customizable in the Sketchatone UI, but I've got some good settings dialed in.
\nFor ultimate control, I've been using a MIDI keyboard to play chords on my right hand and strum with my left.\nBut I also don't want to go too complicated out of the gate while I'm getting used to playing with this. Also it helps at something\nlike an open mic night to keep a small profile, and set up quickly. So for now, the keyboard is out.
\nLuckily, most drawing tablets have some buttons on the side. Using my web based configuration, I can map chords\nto those buttons. The first song that I went really deep on with this is "No Excuses" by Alice in Chains. This one only has 4 chords\nthroughout the song, so really easy to manage.
\nLike I said before, keeping a minimal hardware profile for an open mic night seems critical. There's no time to setup, no mic check - just\nrespect everyone's time and get on and off the stage as quickly as you can.
\nIn addition to the drawing tablet, I need a computer to make it act like a MIDI controller.\nI've been pushing hard on making this work on a Raspberry Pi. And since it's not doing anything\ncomplicated, just pushing MIDI events, it can be a Raspberry Pi 4 which has a pretty low power draw compared to\nthe Pi 5. So, connected to the right device, it can be powered from and pass data through the same USB cable.
\n![\"Raspberry](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2026/05/tabletandpi.jpg\")
\nThe computer that powers the drawing tablet in a custom 3D printed case
I had big DIY plans for this. I actually built a custom Zynthian box. Its great, but at the end of the\nday, I'm not familiar with decent production workflows in general for portable DAWs, so I eventually threw in the towel and bought an MPC One Plus.\nMaybe I'll go back to my Zynthian someday, but I'm really loving the MPC.
\n![\"MPC](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2026/05/mpc.webp\")
So this is what I used! What was great about this is that I really only used two pre-programmed tracks. I think if everything was\nsequenced ahead of time, I'd really just feel like I'm singing over a robot. I want to push on what I can do live as much as possible.
\nObviously drums need to happen. The MPC offers lots of dance/EDM options by default so I haven't explored too much besides the couple\nof normal kits plus the two 80's sounding kits. I started laying down drums with the normal acoustic kit - but I didn't like it just due to\nthe fact that if I want an acoustic drum sound I should just use a real drummer rather than recreate it without any human\nnuances.
\nThat's when I struck gold - hidden in the huge list of drum kits is an 808! The real 808 drum machine was released in 1980 and has been an iconic\nsound since then. Marvin Gaye's "Sexual Healing" and Whitney Houston's "I Wanna Dance with Somebody" are some prime examples. More importantly, it's\nnot trying to sound like a human!
\n\nEspecially with such a beautifully acoustic sounding song like "No Excuses", I don't want to try copying. Instead lets speed ahead\nin exactly the opposite direction. Ultimately, the drum didn't sound as snappy and echoey as I'd like - I wanted to match the original's vibe\nwith this, so I got some advice from AI and added some MPC specific FX (Transient, KillEQ, Maximizer, and Flavor).
\nThe other sequenced track was the bass - using the TubeSynth "Simple fat bass" preset, though I think I tweaked to make it a bit punchier. Honestly, I got the broad strokes from the original in there, but\ndidn't go nuts with some of the more articulate playing from the original. It was probably laziness that got me here, but\nits probably better that way because, again, this isn't a live bass - probably best to not try to be a human here.
\nI used a nice synth patch to fill in for the rhythm guitar. It fills the space super well, and has some nice\nsharp higher ends that bring out the "plucking" of strings that I wanted to highlight with my Sketchatone MIDI controller.\nI used the Jura synth with the "Abandon" preset.
\nI practiced with this for a while, but I started really getting disappointed that I had\nno lead guitar. I was just going to ignore it because sequencing lead guitar parts just feel wrong for this song, missing\nthat human element, and I sure didn't think I could play rhythm and lead simultaneously.
\nAfter spending some time thinking what to do, the guitar solo itself seemed like a lost cause, but the chorus really just needed a bit more volume and\nhigh end, because the singing goes up a register here.
\nI started experimenting with having my MIDI strumming control a second synth track simultaneously.\nThis is when I hit exactly what I needed. I tried overlaying an arpeggiator. I found that\njust normal strumming with subtle variations can really make the arpeggiation feel like an entirely\nseparately controlled thing - making it kinda sound like a lead guitarist!
\nRiding high with this success, I started wondering what I could try with the solo. Obviously, I'd use the\narpeggio again, but wanted some extra noise that I could at least do a noisy grunge discordant thing with.\nSo I added a bendy overdrive synth. It doesn't sound like I'm adding much with normal strumming, but when I use\nSketchatone to start doing some extreme pitch bending while I strum and also dwelling on the highest string, I can start\ngetting kinda Sonic Youth vibes going while still strumming the rhythm part.
\nIts not as interesting as a guitar solo, and is largely just noodling rather than actually\ntrying to articulate notes, but its a good replacement in a pinch.
\nThere's also some nice short lead guitar transitions when the verse transitions to the chorus\nin the original. I did some track automation on the last measure of the verse to let these\nextra lead guitar instruments come in (just unmute the instruments during this part of the verse).
\nAfter some decent practicing (remember, this was the first time I've strummed a song), I thought I sounded\npretty good. I still hadn't solved how I was going to hold the drawing tablet though. Up until two days before\nthe open mic, I just put the thing on top of my music keyboard I had on a stand. Its too hard to hold the tablet\nwhile strumming and operating the buttons. My friend suggested a normal iPad holder for mic stands.
\nThis was a winner! I mounted it to the MPC stand and it worked really well.
\n![\"the](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2026/05/therig.jpg\")
\nThe stand, MPC, and tablet holder in my garage, ready to go!
I'm still a newbie to singing. I'm doing well at home, but really don't have the experience on stage. I've only performed\na handful of times in these past few years.
\nLucky for me, my first open mic was at Smalltown Society. This is a great arts non-profit in\nCastro Valley that really does music well. It is THE absolutely most supportive environment you could ever hope for in an open mic.
\nNormally, I'm a nervous wreck before I do a show, but for this - it was an experiment, and I felt completely open to failure. Of course, I'd rather\nit go well....but I wasn't going to be bummed out if it didn't.
\nI've had the absolute worst luck with stage sound in my tiny music career so far.\nI often can't hear myself and just rely on muscle memory when I sing. My very first time singing\nwas against a backing CD track. It was at the Alameda County Fair in 105 degree weather, where for some reason, the heat messed up the CD causing it to playback 15% slower (and yes pitch was affected).
\nSmalltown though - I'll probably never have better sound. The space is amazing, and the sound engineer Miguel is the best you'll ever work with - a true pro.\nI went in knowing this and even requested some settings I was getting with my vocal effects processor at home.
\nWhat I didn't count on was how much of an absolute diva I guess I am. If I don't sound exactly\nlike a freaking studio recording on stage, it all goes to hell.
\nI immediately thought I was bombing the moment I opened my mouth.\nAs good as the best sound engineers are (Miguel included),\nyou won't sound like a studio recording. The stage monitors will mix with the room, settings will be dialed in during at the start BECAUSE I'M A LOWLY\nOPEN MIC PERFORMER and not doing a stadium with a 30 minute sound check and earpiece.
\nThe "bombing" really threw off my game to the point where I started getting some chords wrong, and made me not want\nto put the audience through any more misery. The third verse was when I really threw in the towel after mis-remembering how it started.
\nIt was only afterwards, when I saw a short clip that I realized that I actually did pretty darn well and it was all in my head until\nI sabotaged it.
\nLesson learned for next time. Stage monitors are to MONITOR - not to be your vocal line out that you can slap on a gold master. Obvious, yes - but I've\nhad only bad experiences so far, so now I know what a stage is really supposed to feel like!
\nI'm really digging the stuff I tried with the lead guitar as a second layer. I have a song\nin mind for next time - maybe it'll work. But I also want to try some more articulated playing\nwith scales instead of chords, so I have some features to add to Sketchatone.
\nOverall, it worked really well. Feels ironic that my weird Frankensteinish tech setup\nworked flawlessly, while my own performance inexperience was the thing that got me!
\nEither way, thanks to the folks at Smalltown Society for making\na really great and supportive space to try new things like this!
\nOh and also for next time, I've ordered some felt nibs for my pen. I had no idea how much it would wear\ndown playing music!
\n![\"The](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2026/05/nib.jpg\")
\nOh no! The nib! This is after just a couple days of practice
Full song recorded from the cozy comfort of home
\n", "date_published": "2026-05-08T00:00:00Z" } , { "id": "https://benfarrell.com/blog/2026-03-23-year-of-ai-coding/", "url": "https://benfarrell.com/blog/2026-03-23-year-of-ai-coding/", "title": "A Year of AI Coding", "content_html": "It wasn't until fall of 2025 that I seriously used AI coding tools.
\nPrior to that, I dabbled, but really never had great results. I was really using AI for the\nstuff I was terrible at like RegEx or shader code - you know the obscure stuff we only do once in a blue moon.
\nShaders were difficult to get right, but a fun thing to try. I'd turn to assorted AI\ntools (mostly free) for RegEx and the like, and it mostly helped, but it was also unusual\nI'd have the problems that annoyed me so much that I'd turn to AI in the first place.
\nIn March of last year my job granted a subscription to Cursor to experiment with.\nI'm not one to shy away from experimenting with new tech, but it was a challenge to integrate into my\nworkflow.
\nRealistically, I've spent 25+ years of my career learning and getting really\ngood at stuff. In my early AI coding explorations, I struggled to see where AI was being competitive to\nwhat I was already doing well.
\nThe easy stuff, writing HTML/CSS/Javascript, I could already do pretty fast. So I tried to use it for the more complex stuff, but it\noften fell into these circular logic traps where it would keep doing the same bad thing over and over again.
\nIt was pretty clear in how I was using it at the time, that Cursor was being more disruptive to my work than actually helping it.
\nIt wasn't until the fall of 2025 that things really fell into place.
\nI'm guessing I started exploring Cursor properly in September of 2025. I had an idea to turn\na drawing tablet into a MIDI controller that emulates playing a guitar. I've just released an\nearly version of this in the past week called Sketchatone.
\nFor this, I needed to explore HID devices and read raw bytes from the drawing tablet. It's something that was completely new to me,\nand I'm not entirely sure where I'd look because I wasn't entirely sure what I was looking for. Cursor, for example, helped me with certain strategies around byte reading\nthat I just wasn't familiar with. Perfectly normal things like big/little endian byte ordering, I hadn't encountered\npreviously (or at least remembered).
\nI was exploring this in Node.js at the time. Javascript and Node.js I'm quite comfortable writing. However, for the sake\nof the project running on various devices like the Raspberry Pi, having a Python version made more sense.\nI've clumsily powered through writing Python before, but it takes me 10x longer than other things because I have\nto second guess and look up virtually everything I do.
\nThis project is where I started heavily playing with Cursor - and it was because it was giving me an opportunity\nto make something with tools I was terrible with.
\nI didn't expect to get good at Python by having AI write code for me. I'm happy to be super good at web tech and have\na number of other languages and platforms that I can reasonably stumble through when I need to.
\nI'm sure I'm marginally better at Python from this experience - but if I truly wanted to be better, I'd have struggled\nthrough it on my own and suffered the failures and rode it through to turn those failures into successes.
\nThis is something that was obvious to me for an entire programming language - but it did sneak up on me in relation to learning\nhow HID (Human interface devices) work.
\nI had gotten into a nice groove of directing Cursor to add a feature, and then I would review the code, and do the commit.\nYes I was being careful with code logic here by reviewing what went in to my project, but I was still letting AI do the thinking\nfor me of integrating these new-to-me concepts around HID devices.
\nWhat happened as a result was that, I learned nothing. I had working code but no knowledge of how it worked on a larger level.\nI could eyeball functions and for loops and see what they were doing on a smaller scale, but HID concepts like "report ID" or "usage page", I had no idea.\nUltimately, I went back much later using AI to summarize all of these things to actually learn what was going on.
\nThis turned problematic. I was still in the stage with Cursor that for higher level complex reasoning I'd often get into circular failures:
\n"Hey cursor, can you add a feature to do X?"
\n"Sure! Here's the code."
\n"But you didn't account for Y. Can you fix that?"
\n"Sure! Here's the code."
\n"But you still didn't account for Y. Can you fix that?"
\n"Sure! Here's the code."
\nAnd repeat....infinitely.
\nI ended up having some harder problems to work through, and no real understanding of how everything worked, while AI wasn't up to the task.
\nWhat I didn't mention was how I used Cursor. I'm a WEBSTORM4LIFE kind of person. I've been subscribing to this Intellij IDE for over a decade, and\ndon't want to let it go. As a result I never got into using VSCode, so as a result Cursor was super awkward for me.
\nMy ACTUAL workflow during all of this was to have Cursor open and just ask it to do stuff for me. I'd ignore the rest of its UI completely, while\nreviewing code and making small edits in Webstorm, which of course flagged all the files that were different from what was committed to git.
\nIn winter of 2025, I was able to get an Augment subscription that I tried out. The ability to have a panel right in Webstorm was REALLY nice and made\nmy workflow feel more connected.
\nSomething was still off though. The big selling point of Augment was the ability to have more context for large codebases. I took advantage of this for my Python/HID\nside projects as well as a C++ project for work (I'm worse with C++ than Python).
\nFor not very complex tasks it was fine, and possibly a tiny bit better than Cursor depending on what I was doing. But it was hard to say until\nI found the other models.
\nYes, I stumble through life without reading the documentation - and hadn't actually tried changing models before. Until now, I just used whatever\nthe default was. But changing to a more advanced model changed the entire game.
\nSuddenly, I wasn't getting the runaround for complex tasks - things were working remarkably well provided I gave the right context in my prompts.
\nAugment was my jam until about a month ago.
\nOn second thought, I might have been switching models back in Cursor after a while. You can only go so long with seeing "auto" in the model\nmenu before you click the dropdown.
\nI remember this because I started using complex models that cost like twice the tokens for using them. I just did this for everything because I\nwas supposed to be "exploring" anyway and my company was footing the bill.
\nSo, a month ago, the worst thing happened: our corporate subscription to Augment lost one of the more advanced models (I think it was Opus 4.6). By this time, these tools were so ingrained into my workflow,\nI was barely actually writing code anymore. Losing this model meant AI was only medium smart now, and I was back to circular arguments again\nto get the more complex stuff done.
\nThe day it went away felt like one of those days that the internet is out, and why bother do work?
\nThis was around the time I started using a shared API key for Claude and Claude Code, which eventually turned into a proper license.
\nAfter all the churn, I have a good handle on what I should use for what. The normal stuff I'm using Augment for. If I'm feeling like I'm getting\ninto an exhausting argument with it or it feels like I probably will, I switch to Claude Code and use the better models.
\nSo that's where I am right now tool wise. As I mentioned before, I was working in Python and C++ as a way to suss out how well AI coding tools\ncould work without me actually coding. This gave me confidence and a good feel on how to be hands off and not code anything myself anymore.
\nEventually this led to me doing all the stuff I'm actually good at with AI too, and it's really made things go pretty quickly. I'm having Augment\nor Claude do a feature for me, and if I have an opinion up front on how to tackle it, I'll give it. That said, I usually just let it do its thing\nand then review code afterwards.
\nIt's led me to be a lot more opinionated than I'm used to. Typically, as a prototyper, I may or may not work with another person on a project.\nBut, if I do, I respect their opinions and don't nag about little things. After all, they know what they are doing and it's just a prototype at\nthe end of the day with a limited shelf life.
\nWith AI tools on the other hand, I'm an absolutely annoying micromanager. I don't actually write the code, but I will complain about every little\nthing it does differently than I would have.
\nSo in that, my role kind of feels like an engineering manager now. With AI coding tools doing the coding, there are some other roles I can resurrect\nwith AI help as well.
\nAs a prototyper, it's not often that I had time for the little things before that made a difference. One of those things is documentation.\nDocumentation is one of those things nobody has time for until you need it, but by then it's a massive undertaking to add it all.\nIt's also one of those things that is standard practice for AI to use as guidelines.
\nLast fall my co-worker introduced me to good markdown AI docs. And by "introduced", I mean had such good results when I was ignorant to it all, that\nI had to pay attention.
\nBut all of these markdown files are really meant to be read by AI. Human readable documentation is such a good thing. I've since leaned heavily into\n11ty static site generators for a few big projects. Not only can AI read the markdown files used here in the source of the static site, but\nit can easily be built as a static site ready for human reading once you publish it. So documentation can really go far if you let it. I originally\nstarted one of my projects with an 11ty docs site just because I thought it could be cool, but it turned into one of those things that I've passed to\na few teams to share my work, went back to understand something I did but forgot how, and even summarized into a context window so an LLM could build what I was building in some experimental calls!
\nSo in terms of engineering practices, documentation is huge. But there's another big one that I'm seeing even bigger value in: automated testing.
\nShould we have been doing automated testing before AI? Yes. Did I do it? No. Do I believe you did it even though it's so important? Probably not.\nTesting is one of those things that engineers are supposed to do, but I struggle to believe most of us practiced what we preached. As a prototyper, it just never made much sense. We slap together code, re-architect constantly,\nand more often than not, throw the project away after a short time.
\nSo, I was always cool with going without tests. It's too much overhead for not enough payoff. But that was before AI coding tools could do it for free.
\nI started in again on unit testing an animation system I was creating. Despite being coded by hand, it kept getting bigger and more complex.\nI had hundreds of tests created, and I didn't write any of them. It paid off big time. This was part of a larger project and was getting too big\nto properly test as a human. I could add features and have a huge safety net that (albeit with some unpredictable holes) without the overhead of\nwriting the tests myself.
\nSince then, I've been leaning into tests everywhere. Not so much for UI, because UI is constantly changing for me, but for the underlying systems\nunderneath the UI. If it's complicated and especially if I keep getting burned by regressions when I manually test it - I just ask for automated tests.
\nOne of the more interesting use cases was for a WebGPU rendering engine I was working with. I used automated testing to verify things visually looked\nhow I needed them to look. It ended up being basically like a 15 minute thing I'd need to watch, but when I was changing the guts of this rendering engine, I'll take\nthe 15 minutes to watch this thing rather than remembering and manually testing everything.
\nThe rendering engine case was a bit harder to setup. But for normal headless testing, it seems super easy to get Playwright tests going.\nI've been more hands off than I probably should be with direction for writing these tests. I've adopted a "pray and spray" approach.
\n"Pray and spray" is coined from first person shooter gaming I think. Basically you just load your machine gun and fire away whether you see your target or not.\nBut if you're familiar with your movie tropes, using AI for testing in this way is kinda like giving a Starwars Stormtrooper a gun.
\nOr going after Predator...so much shooting, and they barely graze it.
\n\nYup, if you don't put in the effort, AI made automated tests can miss pretty badly. I've seen some tests made by AI that just pass without\ntesting anything at all with some complicated code to compensate for it in the assertions.
\nAt that point, the pray and spray approach starts feeling absurd like....
\n\nHowever, this is why it's "pray and spray". I'll have AI make TONS of tests. If I hit a regression that it missed, I'll go back and add more.\nI'll ask why the tests didn't catch something. Overall it seems like a net positive for my minimal effort.
\nI went into AI coding not really excited because it couldn't do much. Now it's a little nerve-wracking because it CAN do too much.\nAt the end of the day, the real bottleneck for my work is going to be human testing. My prototypes are getting more and more complicated\nwith more and more features. Automated testing is only going to save me so much. Features can now be coded fast and not by me.
\nWith AI tooling, I'm more invested in testing what was made after the feature is supposedly done. And that's great.\nBut before when I was coding by hand, I'd test more at every stage of implementing a feature. So the deeper edge case stuff was more battle-ready\nbut the end implementation that held everything together was only marginally tested because I was so bored and ready to move onto the next thing.
\nI think one way to think about this is that you're forced to look at the bigger picture now rather than sweating (or even being aware of) the small details.\nThis can be good and bad. Like I said before, you're not trying/failing/fixing and really cementing those low level details in your head. As a result,\nyou're less likely to innovate solutions to problems with your fuzzy mental model of everything.
\nSomeone mentioned to me that they don't see themselves using AI coding tools because they actually enjoy coding. I thought the same too,\nbut what I think I enjoy more is this bigger picture. The larger systems that can come together for a bigger application. It wasn't\noften I'd get to enjoy this aspect before because those moments were too few and far between because I was slogging through the underlying\ncode before (as much as I do enjoy that too).
\nAnd I think in this way, what we lose with coding by hand, we train ourselves to see the birds eye view more and more.
\nLOL, I'm still not going to read the manual. There's no point. I can't imagine what we're doing today will be anything like what we're doing in 1 year or 2 years.\nI'm going to play with whatever tools I can get my hands on. Honestly, though, the idea we've injected a paid token model as a bottle-neck for work that\nwe've always done with free/open source tools would be a bit dystopian.
\nIt WOULD be dystopian if it weren't for the fact that we already have LLM models that run locally without the internet and it's only going to\nget better. We just have to be a bit careful that we're not inviting these AI companies to become locked into our workflows. Keep exploring, and keep shaking things up\nuntil these tools are our own.
\n", "date_published": "2026-03-23T00:00:00Z" } , { "id": "https://benfarrell.com/blog/2026-03-22-cnced-music-factory-mk4/", "url": "https://benfarrell.com/blog/2026-03-22-cnced-music-factory-mk4/", "title": "CNC-ed Music Factory Mk4", "content_html": "Last October, I blogged about my CNC(ed) Music Factory. It was a good attempt,\nbut I ended up 3D printing the walls, made some mistakes, and while I think it was going to stay together adequately, it wasn't\nas polished as I'd have liked.
\nOriginally, I wanted wooden walls with inset engraved acrylic panels, all done with my CNC. Well, I just finished my 4th (I think) build\nand it's looking pretty nice!
\nOne thing in the last build that wasn't great was that you were plugging directly into the Raspberry Pi in terms\nof power and USB. I wanted this all on a side panel, so I got some connectors and a USB port plate, all I could mount in the\nacrylic.
\n![\"Front](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2026/03/ports-panel.jpg\")
\nFront panel providing power and USB ports as well as MIDI in/out
I'm really digging the blue racing strips I engraved on the walnut wood panel. Unlike the other 4 sides, this is a flat black, non-transparent\nacrylic panel.
\nI really liked having the semi-transparent acrylic on my other one (although it only worked out so well), and wanted to do this up right this time!\nHere's the side panel with 1/4" audio in and out. The engraving is on the back of the acrylic so it shows up as kind of layered\neffect.
\n![\"Audio](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2026/03/audio-io-ports.jpg\")
\nFront panel providing power and USB ports as well as MIDI in/out
This side of the device is also where my USB ports live on the Raspberry Pi. I took advantage of one port and bought a tiny USB LED\nlight to shine through. I was really hoping it would be bright enough to leak light all around the device and light up other sides.\nNo such luck, but where it is looks pretty sweet.
\n![\"LED](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2026/03/lit-up.jpg\")
\nLED Light on side panel
The opposite side is simply a heat vent. I want as much airflow as possible in this little box, so guessed that\nadding this was important. Further down, I cut a little access panel as this is where the SD card lives. At this point\nI'll have to get it in and out with tweezers, but I think it'll do fine.
\n![\"Vent](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2026/03/vent-side.jpg\")
\nVent side
There's some 3D printing shown here. The blue corners cover up the interlocking wall fingers which I like. But on the top, the\nfriction fit sleeve to grip the walls and attach the cover is a bit bulk and ugly. As of now it's temporary, and I want to try\na new solution to grip the cover onto the case. Unfortunately, I think I might need to make a new cover as things aren't lining\nup so well anymore.
\nHere's a top down view of the full cover. Thanks to my friend Carla for coming up with (and doing)\nthe hydro dipping of the cover! I definitely want to keep this as long as I can, and I hope I can salvage this with some tweaks to make this a better\nfit in the future.
\n![\"Top](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2026/03/top.jpg\")
\nTop Down
And of course, this wouldn't be anything if it didn't power on! Here's a closeup of the Zynthian screen with a guitar sound\nloaded and ready to go!
\n![\"Turned](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2026/03/turnedon.jpg\")
\nPowered up with Zynthian screen shown
And lastly, I had to brand this thing. My old friend/bandmate Jay came up with the name CNC-ed Music Factory, and it's awesome.\nSo I engraved it both in the acrylic back panel, as well as the bottom MDF board.
\n![\"Back](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2026/03/back-plate.jpg\")
\nAcrylic engraved back panel
![\"Bottom\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2026/03/bottom.jpg\")
\nMDF Bottom with engraving
All in all, I'm pretty happy. I think the only issue I want to fix soon is finding a better way to attach the top plate\nthat's not so ugly and actually fits better. I also made a mistake with the acrylic insets. My plan was to make them flush with\nthe outside of the box, but I accidentally made them flush with the inside, so there's around a 1/8" rim when looking at it.\nHonestly it'd be pretty cool if you couldn't see the bits of paint I accidentally got on said rim.
\nBut for now, this 4th iteration will do! I also have just been putting some finishing touches on Sketchatone, which runs\non this device using Zynthian, and it's sounding like a great setup especially as I strum!
\n", "date_published": "2026-03-23T00:00:00Z" } , { "id": "https://benfarrell.com/blog/2026-02-23-sketchatone/", "url": "https://benfarrell.com/blog/2026-02-23-sketchatone/", "title": "Project Sketchatone", "content_html": "Sketchatone is the first serious AI coding project I've done. If it weren't for tools\nlike Cursor and Augment, this would be a pretty limited (but fun) experiment. But right\nnow I've got several utilities and installers spread across Python and Javascript, MacOS and two\nflavors of Raspberry Pi.
\nMy motivation was pretty simple - I'm a keys player, but lots of music I like is guitar driven.\nThe strumming adds a nice human richness to the sound, and strumming is easy to do!
\nContrast that with the keys. IMO, you can add lots of richness, but you're doing acrobatics across the keys\nand need to be able to play pretty fast to get those textured fills in.
\nI'll get there, but I also want to do vocals while playing. Strumming a guitar is WAY easier.
\nI was looking around for devices that could do this, and came up shockingly short. My new keyboard controller (Novation Launchkey MK4)\nhas a little strum function, but it's a tiny scroll wheel that's only so useful. Really, I wanted more something like the old Suzuki Omnichord that just relaunched,\nbut as good as this may or may not be, it's $1000!!!
\nIt was tempting! Artist like Rebecca Sugar have made some amazing\nsounding songs on it, but it nagged at me that there should be something more DIY out there.
\nAnd every several years or so, some neat project pops up! Here's an Arduino based one from 2019.\nIt's neat, but for someone who's terrible with electronics, it's a bit much.
\nNope, I'm better with software! And something else has happened over the last decade. Wacom, the company that is known for high-end battery free\ndrawing tablets, had a number of patents expire. This led to a rise of similar, very inexpensive tablets, like these two I bought recently:
\n![\"huion](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2026/02/inspiroy-2-m-pen-tablet-01.webp\")
\nthe Huion Inspiroy 2 Medium ~$70
![\"xppen](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2026/02/xppen.webp\")
\nthe XP Deco 640 ~$30
Why do I mention drawing tablets? Well for one, you can imagine dragging a stylus across the surface and over imaginary strings like a pick.\nDrawing tablets have high resolution and can be very responsive - so strumming action can feel pretty nice here.
\nThe other thing about drawing tablets is that they have a high level of precision in several other areas too! Of course, there's the whole Y axis.\nBut 16,0000 or so levels of pressure sensing is pretty normal for these things.
\nWe also have horizontal and vertical tilt, or basically what angle you're holding the pen at.
\nLastly, most tablets I've seen offer buttons on the side that can trigger various things (though this has been a bit hard to get right if you refer to my Blankslate project post)
\nWe can control a number of things like note velocity, sustain, pitch bending, and even just triggering chord changes!
\nAnd this is all on a device that doesn't need batteries, lots of people have, and can cost as little as $30!
\nLate last year, I had a good first iteration of this. But honestly, AI tools helped me build it a bit too quickly. Using HID (human interface device) APIs\ncan be super finicky and confusing when interacting with a device that your OS tries to lock down, but also when the manufacturer really would\nrather you use their driver software and not go mucking around with it yourself.
\nThe result was that AI helped speed me through building a good first prototype, and I thought I understood how it all worked. But things kept changing under my feet. I had built a light configuration\ntool to help "learn" the bytecodes and ways to interact with various tablets, but given how AI "helped" me, I didn't really understand some of the nuances.
\nSo it was flaky, and all the low level tablet interaction code was wrapped up in my project code. So I broke it out into Project Blankslate and made that\nproject a dependency of my new strumming project.
\nI discovered and learned all sorts of ways HID devices can act. They can act one way, but then on separate interfaces altogether when the driver software takes over.\nThey can have keyboard interfaces either through their drawing interface or separately - which matters, because your OS REALLY likes locking down keyboards.
\nI also learned about WebHID, so we can strum in the browser and send MIDI with nothing installed! But as you can read in my Project Blankslate post, there are some\npretty big limitations we can't overcome with a browser.
\nIn the end I think I made Blankslate a pretty solid tool for reading tablets and generating configurations to read them. Although with only having tested my 2 tablets, who's to say!
\nAI did really help me here, because I'm left with a Python version, a Node.js version, and a WebHID version. I've got tons of unit tests, and some pretty robust documentation. Doing all of this\nwithout AI help would have taken a LOT longer!
\nSo, strumming on a tablet connected to my laptop is one thing. I can watch the notes and see that "YES it's doing the right thing in theory", but at some point, you gotta start playing some music.\nThe obvious first try is just to get this working locally on a DAW (digital audio workstation).
\nFortunately, I have Ableton Live 12 installed! This was an easy setup. Get the Sketchatone running, arm the MIDI track in Ableton for recording, and go!\nIt worked well, and was the primary way I tested things on this project to get the strumming feeling right and all the features feeling like a real instrument.
\nTapping the tablet buttons are fine for chord changes - it can act a lot like the Omnichord in that aspect. Press a button to change to a E flat minor, and now you're\nstrumming an E flat minor.
\nBut the fact is that I'm pretty good with chords on a keyboard as well. So allowing MIDI in to change the strum chord felt really nice.\nI had my drawing tablet on my left and my keyboard on my right (I'm actually left handed, so it worked out great), I could strum a rhythm on my left hand and do all\nthe chord changes on my right. Also, I've made a lot of aspects of Sketchatone configurable, and this includes MIDI channels and devices.
\nSo, what I ended up doing was to send my MIDI keyboard controller to BOTH Sketchatone for strumming AND to Ableton. Sketchatone would control a plucking\ninstrument on one MIDI channel, while the keyboard controller both controls the Sketchatone chords and plays some backing block chords with some luxurious\npads on another channel.
\nIt made a pretty darn full sound!
\nNow that's my laptop. What would be nice is to make this a little more rock and roll - no bringing out a Macbook Pro on stage.
\nI've been exploring a nice little project called Zynthian. Zynthian is a environment with amazing synth and instrument plugins, effects, and a sequencer and clip launcher.\nIt's also built with a custom Raspberry Pi OS. Not wanting to commit and buy a premade kit, I loaded up my own Raspberry Pi with the Zynthian OS and built my own case. I also added a PiSound hat to get some low latency audio as well as MIDI and instrument\nins/outs. I blogged about this here.
\nWith a little help on their (extremely nice and amazing) forums, I was able to get this project showing up as a proper\nMIDI device in the Zynthian UI. And this was essential for using "multitimbral mode" where I could assign different MIDI inputs to different channels. In fact, I'm remembering now that the idea to make this into a service rather than a Zynthian\nplug-in was from this same thread. Good advice - because it can work lots more places now!
\n![\"Sketchatone](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2026/02/zynthian-midi.jpg\")
So the Zynthian is a nice non-laptop based place to try this, but I had another Raspberry Pi 4 laying around. I 3D printed a case for it off of Thingiverse. The plan: to use it as a standalone MIDI device\nthat uses MIDI out and allows strumming on a normal MIDI keyboard! After lots of struggling with dropped MIDI notes, it finally works! If you can live with the 30 second boot time for a tiny device that looks like it should just power on, it's kind of awesome.
\n![\"Sketchatone](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2026/02/pimidihost.jpg\")
Being a Raspberry Pi with onboard wi-fi, however, we can take advantage of it being a mini computer and do something that works on my laptop as well.\nAnd that is run a server!
\nOne thing I haven't mentioned yet is the effort I put into UI on this project. Yes, it operates headless. In fact, fingers crossed, hotplugging different drawing tablets should work well provided the configuration files had been generated and added to the devices folder.\nBut also, there's so many different ways to strum! Even if there was one "right way" that works for everyone, I'm sure I still don't know it yet.
\nThis is why I have a web based configuration tool. Whether on the Raspberry Pi, Zynthian, or laptop, Sketchatone fires up a web server which connects over websockets to the core service.\nIt shows you interacting with the tablet live, but more importantly allows configuration. For example, note duration or pitch bend can be configured to use the tilt direction or the Y axis with settings to adjust the input\nalong a curve or just linear mapping.
\n![\"Sketchatone](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2026/02/dashboard-parameter-mappings.png\")
Strum release is another thing I added. At that point I was just listening to acoustic sets and hearing percussive thumping on the strings, so added a feature that when on and you lift off the strum, it can send an event to another\nMIDI channel. I think it should be percussive - but you do you!
\n![\"Sketchatone](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2026/02/dashboard-strum-release.png\")
I also consolidated a number of features an "action" system of sorts. Transposing up or down an octave (or semitones) is a nice thing you should be able to do live. Maybe allowing a note repeater is something you do or don't want, and you\nwant to turn it on at a moments notice. Chord switching between a C Major pop progression is nice too.
\n![\"Actions\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2026/02/dashboard-action-rules.png\")
All of these things can be triggered by a button press on the tablet. And there's a customizable actions panel to do the assignments. This is a recent addition after some last minute refactoring - so fingers crossed it works well, but I'm been doing some chord switching and strumming\non my Huion tablet with real physical buttons....it feels really good!
\nLike I said, I've been using my XP Pen Deco 640, and the Huion Inspiroy 2 Medium. I absolutely haven't tried this project on your tablet if it's not either of these.\nBut, IN THEORY, my configuration tool on Project Blankslate should work.
\nI've actually found that the despite the annoying limitation of WebHID not allowing you to stop using your tablet as a mouse cursor without some special tricks outlined in my WebHID post,\nit can actually reasonably configure the digitizer part of your tablet pretty well. This means drawing functionality, pen pressure, tilt, all can be configured pretty smoothly. In fact, from the two devices I have,\nmy guess is that there won't be too much difference between tablets in this regard.
\nThe buttons on the side of the tablet are the painful bits. WebHID just completely blocks the HID keyboard unfortunately, so to properly run the wizard for keyboard events as well, you'd\nneed to use the Python or Node.js versions (both should work the same if I've tested properly).
\nBut if you've managed to generate a JSON config file, this can be added to Sketchatone's devices folder and will be autoloaded when the device is plugged in!
\nI will say if you have a config that you made that works well, I'd love to add it and get a library going! Maybe a PR to my repo would be nice!
\nTBH, I was kind of obsessed with getting this done. So my plan is just to let this simmer and play with it! I definitely plan to fix things as they come up.\nI'll probably think of new actions to add to the actions system, but I don't want to keep coding this for the sake of coding it.
\nTHAT SAID, I was just thinking the other day that there are other types of expressive instruments this could emulate. The one thing that comes to mind\nare brass instruments. A slide trombone, for example really can't be emulated well with a keyboard and pitch wheel. You'd have to exactly control the timing\nfrom note to note, coordinating with the pitch bend to make notes glide into each other. A drawing tablet experience could really control the pitch between notes\nextremely accurately. So future explorations will likely be different control mechanisms besides the strum!
\nAll that said - I've really just made another instrument. And instruments take time to get good at, and having never played guitar, I'll have to work on my strum technique!
\nHere's all the docs where you can find instructions, releases and more!\nhttps://sketchatone.com
\n\n\n\n", "date_published": "2026-02-23T00:00:00Z" } , { "id": "https://benfarrell.com/blog/2026-02-16-blankslate/", "url": "https://benfarrell.com/blog/2026-02-16-blankslate/", "title": "Project Blankslate", "content_html": "In my last post about WebHID and Drawing Tablets, I talked about experimenting with the\nChrome/Edge only API for reading HID (human interface devices) from the browser. I also mentioned my endgame for my exploration.\nI was exploring using a drawing tablet as a sort of MIDI controller. I'll get into this in my next post,\nbut in my first iteration of this project the drawing tablet reading was part of it. But as things got more complicated,\nit grew bigger and harder to manage. So I broke out the drawing tablet code into it's own project.
\nSo that's what I'll talk about in this post! Blankslate is two main tools. One tool reads raw HID bytecodes and turns them into\nactual tablet events that can be used in a project.
\n![\"reader](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2026/02/blankslate-cli-reader.png\")
\nOne of the Blankslate readers (Python and Node.js versions available)
The unfortunate reality, though, is that drawing tablets have different implementations from different vendors.\nI get into this in my last post in detail through the lens of WebHID and this means a simple tool to read drawing\ntablets isn't so easy.
\nI've explored two different tablets so far. One is the the XPPen Deco 640.\nHonestly, I bought this one because it was $30, so pretty inexpensive! It offers the normal drawing stuff - X, Y drawing coordinates, lots of pressure sensitivity,\nstylus tilt reporting, and a primary and secondary button on the pen. It even allows differentiation on coordinates when the pen is hovering vs drawing.
\n\nthe XP Deco 640
In this way, only small details seem to differentiate between this one and my new Huion Inspiroy 2 M.\nI picked up the Huion to be another option to try, but also because it has physical buttons on the side that you press, unlike the Deco which has virtual\nbuttons that you tap (and really have to look close to see as you use them).
\n\nthe Huion Inspiroy 2 Medium
The Huion also has 3 different modes with 8 buttons each plus a scroll wheel. This really makes 30 buttons compared to the XP's 8!
\nBut this is also where the tablets diverge A LOT. The Huion uses a keyboard HID interface for this, while the XP's buttons are nicely on the digitizer interface.
\nCouple this with the fact that the driver software will actually make the tablets operate slightly differently when it comes to interacting with the byte data.
\nSo it's complicated!
\nThis is why in addition to the reader, we have a configuration tool.
\nMaybe I rabbit holed too much on this, but I made a configuration tool where a user performs specific actions with the tablet, and based on these actions, the software learns\nhow to parse the bytes, which bytes to read, and how to turn this into actual tablet events we can consume.
\nFor example, step one asks the user to draw across the tablet horizontally, and step two - all the way up and down. This shows which bytes are used for the X and Y coordinates and allows\nus to see the bounds of the data as we get to the edges.
\n![\"configuration](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2026/02/blankslate-cli-config.png\")
\nThe step by step configuration tool (Python and Node.js versions available)
Even though I said that the actual drawing stuff seems like it might be similar from tablet to tablet, which interface, model, and report ID are all recorded\nto help us connect to our specific device later, when we want to target a single one we've just plugged in.
\nIn the end, we end up with a JSON configuration file that has all the details for turning raw byte data read over HID into tablet events.
\n\"x\": {\n\t\"byteIndex\": [2, 3],\n\t\"max\": 32767,\n\t\"type\": \"multi-byte-range\"\n},\n\"y\": {\n\t\"byteIndex\": [4, 5],\n\t\"max\": 32767,\n\t\"type\": \"multi-byte-range\"\n},\n\"pressure\": {\n\t\"byteIndex\": [6, 7],\n\t\"max\": 8191,\n\t\"type\": \"multi-byte-range\"\n},\n\"tiltX\": {\n\t\"byteIndex\": [8],\n\t\"positiveMax\": 80,\n\t\"negativeMin\": 176,\n\t\"negativeMax\": 255,\n\t\"type\": \"bipolar-range\"\n},I don't know if I would have even attempted this project if it weren't for AI coding tools. There's too much I didn't know - even down to using "little endian" byte reading. Cursor, and later Augment, helped\nme explore all the bits here that I was over my head on. I was also a little wishy-washy on languages.
\nMy first iteration on this was using Node.js and the node-hid package for device reading.\nPartway though I discovered that Chrome launched WebHID in 2021. Given that it's all Typescript/Javascript anyway, I implemented both ways.\nI covered CLI tools in Node, browser only WebHID, and even using a websocket server in Node to message the tablet events to the browser.
\nBut even then I wanted this to work on a Raspberry Pi, so Python was the best choice since it's already installed by default.
\nSo now, I'm maintaining three different implementations, with multiple flavors of each. I'm also managing a web dashboard and CLI interfaces for reading and configuring!
\nThere's no way I would have expanded my scope so much without AI coding tools. I've also learned that expanded scope means more bugs and regressions - so I also\nlearned to lean into AI generated tests. I haven't used automated testing in a while, but now for stability on such a big surface area, it feels like a must!
\nIt also enabled some comprehensive (maybe too comprehensive) documentation, which I just wouldn't have had the patience for previously.
\nAnd honestly, this final step of blogging helps my mental model and understanding - because quite honestly, before I started writing, I only vaguely understood the ins and outs of\nbyte reading and device interaction with HID!
\nI was also able to go all out with some web components and Lit.dev for a decent looking dashboard app. It shows events coming in,\nand SVG graphics of the tablet in action, so we can have some immediate feedback the device reading is doing what you think it's doing.
\n![\"dashboard\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2026/02/blankslate-screen.png\")
\nBlankslate Dashboard
And of course, the dashboard also has a configuration tool to help you learn your device and create a configuration file for it, just like the Python and Node CLI tools.
\n![\"dashboard\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2026/02/blankslate-config-tool.png\")
\nBlankslate Config tool
For now, I've made exports available via my package.json for the visualization components and JS utilities.\nFor Python, I let my AI coding tools help out with exporting as a dependency from the repo. Like I said at the top, in addition to learning,\nthis is all in service of a nifty little music project I'm trying out. I'll likely publish to NPM soon as I know the Node/JS side well - but for now\nboth Python and JS seem to work well enough as aliases.
\nFeel free to check out the repo! Again, its kind of a big project put together in a shorter time, so while I have some (fingers crossed) decent automated testing here - it might be brittle\nin the deeper corners.
\nhttps://github.com/bengfarrell/blankslate
\n\n\nLastly, in my next post I'll tell ya all about "Project Sketchatone"!
\n", "date_published": "2026-02-16T00:00:00Z" } , { "id": "https://benfarrell.com/blog/2026-02-15-webhid-and-drawing-tablets/", "url": "https://benfarrell.com/blog/2026-02-15-webhid-and-drawing-tablets/", "title": "WebHID and Drawing Tablets", "content_html": "A while back I had an idea for a project that might benefit from a drawing tablet. I'll brag on this later\non my blog when I've polished and played a bunch more with it, but exploring how to interact with it\ngot me sucked into the world of HID (human-interface-devices).
\nAs I was struggling through reading the byte formats with the help of some AI assisted coding tools,\nI happened to glance over at caniuse and saw that Chrome actually supports\nHID (WebHID) as of 2021!
\nYes - we can read HID devices natively in the browser! Little did I know at the time, that drawing tablets\nare maybe the absolute WORST place to start learning this stuff.
\nBefore I go there, lets talk the easy stuff - just getting permissions!
\nFirst, we'll request device access:
\nconst devices = await navigator.hid.requestDevice({\n filters: filters || []\n});Obviously, given the await keyword, this is an async call. But, more importantly, you can provide an\noptional filters argument. We can filter by things like vendor or product IDs. These will\nalways come with the device handshake.
More importantly too, we can filter by usagePage. Basically, this is an identifier for what TYPE\nof device we're connecting to. So for example if we wanted to connect only to "digitizers" (what a drawing tablet is),\nwe could filter by usagePage: 0x0D.
Filtering or not filtering really depends on your use-case. If you don't really know what you want, you'll get a long list\nof various devices you can connect to, and it can be confusing. On my Macbook Pro, for example, I can see a number of HID devices\nwithout even connecting anything! This is because I can see the trackpad, keyboard, and other built-in devices.
\n![\"webhid](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2026/02/webhid-picker.png\")
But if you already know what device you want to connect to, because you've explored it before, and know the the vendor, device, ID, or even the TYPE of device, then\nit can be much easier to filter and get exactly what you want.
\nThe requestDevice call really only launches the picker, however. This call provides a way to approve device connection. It puts any device you approve\nin a bucket of devices that you can then connect to.
If you've approved a device in a past browser session, requestDevice isn't needed. You can simply auto-connect because permission has already been granted.
Either way, whether you skip requestDevice or not, you'll next want to get a list of the devices you have permissions for:
const myDevices = await navigator.hid.getDevices();How these are called and the user flow is really up to you. In my project, I auto-connect first. If the device\nI want is found, then I open it - the user never sees the dialog to approve anything.
\nOne UX hole in this flow is if a device you've previously connected to exists, but you actually want another. In my project,\nthis likely won't happen based on what I expect users to do.
\nPoint being, though, you'll have to plan how to use these calls for your specific web app.
\nFrom here, you can simply open the device (or devices) you want with device.open().\nEasy, right?
Well, there's a tiny bit more to it, and that's actually receiving data.
\nIf you don't want to miss a single event, add this next event listener before opening, but honestly, you'll\nlikely not be impacted by missing a few events during this short timeframe.
\ndevice.addEventListener('inputreport', (event: HIDInputReportEvent) => {\n\t...\n});Other than listening for errors, this is the only thing you can listen to on the device object itself.\nI will mention that you can listen for a user physically plugging in or unplugging devices through
navigator.hid.addEventListener('(dis)connect', (event) => {\n console.log('Device connected:', event.device);\n});For the HIDInputReportEvent, there are only 3 properties we need to care about.\nThe event returns the device that the event came from. Of course, this can be useful if you're listening to multiple\nopen devices and just want to use a single callback function.
The next thing you can get from an event is the reportId. Basically, a single device\ncan operate in multiple "modes". For example a drawing tablet MIGHT operate as a single\ndevice when using the stylus vs using some of the physical buttons they have on the side.\nFor this, we can use the reportID to help us figure out which HID device
One thing that's nice about WebHID, is that the reportID is given to you as a number. For other\nruntimes like Python or Node.js, you'll have to parse this out of the data yourself to know.
For reading the actual data, a Uint8Array is created from event.data.
device.addEventListener('inputreport', (event: HIDInputReportEvent) => {\n const dv = event.data;\n const bytes = new Uint8Array(dv.buffer, dv.byteOffset, dv.byteLength);\n});Apparently the byte offset can be important because the data can be part of a larger\ndata buffer, and we need to know exactly where the data starts to populate our array.
\nBut from there, we have an array of byte values to work with (typically 10-12 bytes depending on the device). Different actions on the drawing tablet\nwill change these values in different ways and its up to you to figure it out!
\nInterpreting is where it gets a little tricky! So far, I've tried two drawing tablets - but beyond\nthat I'm just spending more and more money to accumulate drawing tablets to make assumptions on how they work and if they\nadopt the same byte formats for similar actions.
\nUntil then, here's a little snapshot of what I've learned for a Huion Inspiroy 2 Medium and a XP Deco 640:
\n\nthe Huion Inspiroy 2 Medium
\nthe XP Deco 640
When I say "little endian", this means that you use 2 bytes at the same time to come up\nwith one number. The first byte is the "low byte", and the second "high".
\nThe x coordinate would therefore be:
\nconst x = bytes[1] | (bytes[2] << 8);We're taking the third byte, shifting it 8 bits to the left, and then doing a bitwise OR to combine the\nresults.
\nYes. I asked AI how to do this for me.
\nIn easier to understand terms, it would be:
\nconst x = bytes[1] + (bytes[2] * 256);For tilt, I found:
\n0 = center (0°)
\n30 = halfway positive tilt
\n60 = max positive tilt
\n196 = max negative tilt (wraps from 255 side)
\n225 = halfway negative tilt
\n255 = slight negative tilt
\nSo it kind of reads normally for positive tilt, but negative starts at 255 and counts down.
\nBut again, is this all HID digitizers? I don't know. There's not too much reason to go into details here when\nI just found it works this way on my two devices!
\nThis, however, is why I created a project I'm calling "Blankslate". To create a workflow\nto have a user send this events in a consistent way, and have a user experience "wizard" of sorts\nthat learns how to map this raw byte data into a consistent stream of tablet events.
\n"Blankslate" generates a JSON configuration file for your tablet, including what device interface IDs to listen to,\nwhat report IDs to expect, and how to parse the bytes.
\nIf you want to find out more about this experimental project, check out Blankslate!
\nTo close this post out, I wanted to go back to what I opened with.
\nLittle did I know at the time, that drawing tablets\nare maybe the absolute WORST place to start learning this stuff.
\nAnd this is in addition to the fact that WebHID is Chrome/Edge only.
\nThe reasons are a mix of sandboxing and OS restrictions.
\nFirst, at least on MacOS, a standard digitizer is hard wired on the OS level to\ncontrol your mouse cursor. There is absolutely no way to turn this off at the OS level\nfrom what I can see.
\nIn Python or Node.js however, you can get "exclusive" access to the device. Basically, when you open it,\nevents are no longer sent to the OS and you can just interpret them.
\nWith WebHID, there's no such luck. I guess it's probably some kind of security issue to\nallow a website to take over your devices? I'm not sure.
\nThis means, in a browser, you're at the mercy of your device driver settings to help you out.\nWith the XP Pen Deco 640 software, I didn't see any way to stop this from happening.\nIt's a shame because I really want to use my drawing tablet as an alternate input source,\nand having it control the mouse is distracting to say the least.
\nI did find a quirk though! If I start the driver software, this software now gets exclusive\ndevice access. Your OS stops listening while the driver is now managing your mouse cursor. If you\nexit the application, the OS doesn't actually take over again until you unplug and re-plug the device back in (or resart your computer).
\nYou've just cheated your way into getting exclusive device access!
\nThe other reality is if your tablet comes with buttons on the side. For my XP Deco 640,\nthese buttons actually fall under the digitizer interface IF you don't have the driver software running.
\nSo if the driver software isn't running, you'll get button events as keyboard events by default, but you\ncan ALSO tap into the raw HID events and get them as bytes as well.
\nWhen the driver software IS running however, these get changed into keyboard events only as the driver software\ntakes over.
\nOn the Huion Inspiroy 2 Medium, I actually haven't downloaded the driver software yet, but without the driver\nthe buttons are actually NOT part of the "digitizer" HID interface at all, but on a separate Keyboard HID style interface.
\nUnfortunately, Keyboard HID events are not available through WebHID - they are actually banned under a security blocklist!\nIt sounds like allowing a web app to listen to keyboard events are enough of a potential security (spying) threat\nto just ban that entire category of device (usage page).
\nEven in Python or Node.js on MacOS you need to run as root (sudo) to get Keyboard HID events.
\nApart from Blankslate mentioned above as a tablet configuration generation tool, I'm working on another\nproject that uses all of this to use a drawing tablet as a MIDI controller of sorts.
\nGiven all the limitations with drawing tablets and WebHID, the browser aspect will likely\nbe more of a novelty and the heavy lifting in Node.js or Python.
\nBut overall, WebHID seems like it could be pretty cool for connecting all sorts of\nother devices that your OS doesn't have strong opinions about. I'm excited to explore\nNintendo Switch controllers over bluetooth/WebHID next!
\n", "date_published": "2026-02-15T00:00:00Z" } , { "id": "https://benfarrell.com/blog/2026-02-01-emf-microphone/", "url": "https://benfarrell.com/blog/2026-02-01-emf-microphone/", "title": "EMF Microphone", "content_html": "One of the smaller experiments I've been wanting to try lately is listening to electromagnetic fields. Inspiration has come mainly from the Japanese group\nElectronico Fantasticos. I've seen some rad experiments come out of folks that are inspired by them as well - mainly inspired\nby this CRT TV Guitar project:
\n\nThere are certainly some cool EMF related projects that have been done and could be done, but I don't have much experience with just listening to how EMF sounds when you aren't trying\nto manipulate the fields.
\nSo, I wanted to try some EMF field recording! But of course I needed a microphone.
\nIt turns out that EMF microphones are super simple. Basically you just attach some copper wire to some headphone wires. The input needs to go through some\nkind of signal amplifier, and then out to some headphones.
\nI initially started with this DIY instruction guide, but embarrassingly enough, I ended up\npartially melting my battery pack. Sigh. I'm not good with electronics, but I think my sloppy solder job made a bad 6v battery loop...and things just got hot. Oops.
\nBut then I asked myself why I bothered to buy a mini signal amp at all. I have both a mini audio amp, as well as a nice Zoom H2N microphone that amplifies incoming external mics already.
\nWith a cheap camera cage I already had, I assembled this beast!
\n![\"EMF](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2026/02/rig.jpg\")
That blue, purplish thing at the end? That's an entire spool of copper wire! I decided to get a bit fancy with this as an excuse to use my 3D printer again.\nSo I kit bashed a wire holder from Thingiverse (with a cute little cap you can remove), with a microphone attachment I also found on Thingiverse.
\nI went through an iteration or two before I had something that my copper wire fit into - but once I finished, it was time for field recording!
\nWhen this microphone was freshly built, I got some interesting results, but it was from electronics I knew would be noisy because they're dealing with\nwireless signals.
\nFirst I tried my Nest thermostat:\n![\"Nest](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2026/02/nest.jpg\")
And next my Macbook pro:\n![\"Macbook](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2026/02/laptop.jpg\")
That was when I thought i'd take it with me to do some "field recording". This actually wasn't all that interesting. As I passed electric boxes in the park,\nor christmas lights hung around, I only got low level buzzing sounds. Yeah, probably the constant stream of unchanging current, or being really well shielded\nmade things pretty quiet.
\nEven the CRT TV I found in an arcade with Rampage on it, was just....OK.
\n![\"Rampage](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2026/02/videogame.jpg\")
I think the electronic boxes you find outside in parks are the most disappointing. I expected some good noise - but I just got a low hum. I think this\nproves I'll need to be more discriminating with my field recording from now on, or even just find stuff around the house. Because well shielded things\nthat just have electricity pass through just aren't interesting!
\n![\"Rampage](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2026/02/powerbox.jpg\")
I'll probably try some more field recording - but I don't think me just taking a walk around the neighborhood is going to net me anything\ninteresting. I'll probably even have to setup some noisy scenarios. Either way, I think there's room to get some interesting sounds after a first\nventure out!
\nAlso, there's a fancy new design I found on Github. I have some spare acrylic and a CNC machine, so I want to\nput this together real soon and see if I hear any difference!
\n", "date_published": "2026-02-01T00:00:00Z" } , { "id": "https://benfarrell.com/blog/2025-10-03-cnced-music-factory/", "url": "https://benfarrell.com/blog/2025-10-03-cnced-music-factory/", "title": "CNC(ed) Music Factory", "content_html": "I JUST finished a build of a custom case for my Raspberry Pi based music box. Currently, it's based\non Zynthian.
\nZynthian sells hardware kits, but offers DIY Raspberry Pi boot loaders so you can flash an SD card, plug\nit in to your Pi and GO!
\n![\"zynthian\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2025/10/zynthian.jpg\")
\nThe Zynthian kit, after assembly, if you purchase it rather than custom build it like I did....
I say "currently" because I bought a PiSound a few years ago thinking\nI'd build out a kit with their PatchboxOS on it.
\n![\"PiSound\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2025/10/pisound.jpg\")
\nThe PiSound is a Raspberry Pi hat that supplies digital audio conversion with 1/4" in and out, as well as MIDI in and out
Zynthian IS rad, and it seems like the complete package: it comes with a step sequencer, MIDI recorder,\nand just feels as close as we can get to a digital audio workstation in a small form factor on the Raspberry Pi.
\nMaybe I'll find something better than Zynthian someday? And it wouldn't matter because I could just boot that onto my\nRaspberry Pi instead. It won't be soon though, because the Zynthian community is so helpful and vested to get you set up.\nI've also written a couple of half-assed device drivers in Python for my Akai drum pad and keyboard.
\nPoint is... Zynthian is open, friendly, and just plain exciting. But I'm used to being burnt by tech, so I'm playing it safe. If\nthis doesn't work out, I'll just load up something else on my Raspberry Pi.
\nAs much as I love learning and playing with these types of projects, I do actually have a goal.\nOver the past few years, I've come back to playing music. I'm getting better as a singer, and incorporating\nkeys into it better too.
\nIt turns out, though, that I really appreciate a fuller sound. I need more instruments. And some bass. And someone to keep my\ndamn time for me! This is normally when I'd just join a band, but I'm all over the place right now.\nI told my friend about my aspirations into doing synthpop glam rock covers while ALSO agreeing with my music\nteacher that "yes, I should really get up to speed in the blues".
\nSo yes...for now, I need to get my shit together and figure out what I actually want to play.\nThe Raspberry Pi Loaded with Zynthian is majorly going to help me play, experiment, and hopefully open-mic.
\nI'm really against the idea of going on stage with a laptop. It's so not punk rock.\nBut something that has the same form factor of a guitar pedal, but as powerful as a computer from\n10 years ago, and I built it - so it's janky as hell....
\nWell, maybe that's not punk rock either, but it's SOMETHING NEAT.
\nLike I said, I'm not going to be sold on Zynthian until I put it through its paces. I so appreciate Zynthian being open-source\nand allowing me to flash an entire OS image to my PI!
\nThat said, it turned out to be way more work (and actually more expensive in the end)...because if I'm gonna do this, it's gotta\nbe punk rock leaning (or at least just SOMETHING NEAT).
\nAnd that leads me to when I bought my first CNC machine.
\nPrior to a year ago around this time, I had access to a laser cutter. It was pretty neat what you could\ndo with cutting and engraving wood and acrylic as I wrote about previously.
\nI even made a few Rasperry Pi + PiSound cases before. The problem is that I wanted to add a screen this time and there was no\nplace for it on the case designs I got from them.
\n![\"Custom](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2025/10/screenless.jpg\")
\nThe first (screenless) case I built with help from a laser cutter
After I started playing around with Zynthian, I knew it was something I wanted to actually use\nvs just playing around with.
\nUnfortunately at this point, I had no laser cutter access anymore! Despite our town of Hayward\nhaving a library makerspace with one, I knew that it was going to make lots of mistakes, so not having\nconstant access and having to schedule time would likely kill all momentum on this.
\nSo, I bought a CNC machine off Amazon. The Sainsmart 4040 Pro\nis probably on the edge of cheap/beginner entry vs a solid machine. I paid a bit extra for the pro because\nI read about not having belt driven axes (they tend to wobble I guess?).
\nAll in all, it's pretty fun to use, albeit a steep learning curve (true of any CNC)! I think if I doubled what I spent\nI'd probably end up with loads more speed and power. I don't NECESSARILY like playing a game of chicken\nwith my cuts (either they take an hour or two or I break my bit), but for one-off projects like this, it's totally fine!
\nUh oh. I've set some things in motion. Cut to my weapon forging montage.
\n![\"the](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2025/10/mandy-forging.jpg\")
\nNic Cage forging "The Beast" in Mandy
Back when I was using the laser cutter/engraver, we quickly settled on using LightBurn.\nThough its a bit expensive (between $100 and $200), it's likely the best and most complete tool available.
\nThe world of CNC has different options. Like laser cutting, the free software tools are pretty finicky and hard to use.\nI started out with Universal G-Code Sender. Using this meant of mix of me learning some\nactual GCode commands and dealing with a potentially buggy and somewhat difficult to use design view. Despite the learning curve,\nit also forced me to learn a fair bit of how CNC machines ACTUALLY work.
\nOther software, like Easel are subscription based. Though Easel proved pretty nice throughout it's trial, I really\ncan't see me paying a monthly subscription when I only plan on using my CNC sparingly and in spurts.
\nFortunately, Lightburn has a pre-release candidate for their CNC software called Millmage.\nI'm still in my trial period, but I definitely plan to buy the (non-subscription based) license when it's released.
\n![\"sample](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2025/10/samplecut-yellow.jpg\")
\nAn early cutting attempt...
![\"sample](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2025/10/testcut-red.jpg\")
\nAnother early cutting attempt, yes...I'm bad at measuring stuff
One of the things I really dug about semi-transparent acrylic was that I can engrave on the back and it looks pretty dang rad.
\n![\"engraving\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2025/10/engraving.jpg\")
\nEngraved piece with an AI generated swirly pattern that I made in Adobe Illustrator
Once I blew through 10's of dollars worth of acrylic sheets and finally had my measurements right, I was able to assemble a test case.
\n![\"engraving\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2025/10/first-assemble.jpg\")
\nThe first fully assembled pass of the CNC(ed) Music Factory
You'll notice the rather rough looking wood top plate here. First of all, my choice of wood was probably a bit too soft, so I ended up\nswitching to walnut which worked quite nicely. I also don't have that great of a handle on engraving yet - especially if I want it deeper, like here.\nThe carving bit gets thicker toward the top, so it's not so great at finer details as it plunges more. Plus that gold paint to fill things in\nwasn't the best either.
\nLike I said, big learning curve (that I'm still involved with), but once I knew I had the dimensions right, the last cut was exciting\nand suspenseful!
\n![\"Final](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2025/10/cutting.jpg\")
\nFinal cutting pass
![\"Final](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2025/10/cutting-closeup.jpg\")
\nFinal cutting pass (close-up)
In the end, I took my Raspberry Pi + PiSound + a ribbon cable to connect the two that I had mounted onto acrylic with screws and spacers...
\n![\"Naked](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2025/10/nocase.jpg\")
\nThe naked CNC(ed) Music Factory before the case is mounted
...and added a totally badass looking case. I had some help with the front panel. My friend Carla setup a makeshift hydro-dipping station in her\ndriveway. The spray paint pattern she laid down makes this really come together looking like a proper guitar pedal, I think!
\n![\"Finished\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2025/10/finished-full.jpg\")
\nThe finished product (top-view)
![\"Finished\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2025/10/finished-side.jpg\")
\nThe finished product (side-view)
Now, of course I had to give this a name. Unfortunately, just as I ran out of blue acrylic, I noticed I engraved\nupside down. Oops. Something to fix on my next iteration.
\n![\"Finished\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2025/10/back-label.jpg\")
\nA mistakenly upside down back-panel
And with that, it's done!
\nNah, it's probably not REALLY done. I'll probably make improvements. If I have to cut that back panel again, I'll\nbe sure to engrave right side up.
\nI'm also a bit shocked at how much it's not falling apart right now - and in fact really stable! The sides and top are all separate pieces held together\nby tiny 3D printed corner pieces. I initially made them to hold the 1/8" thickness of the acrylic, but they slid out fairly easily.\nI reprinted the ones on top a little narrower so you'd really have to jam them in and they hold tight.
\n![\"Corner](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2025/10/3dprinted-corner.jpg\")
\n3D printed corner joiner shown in Blender
I also added some stick-on rubber feet to the bottom to cover my screws and elevate it a little. Plus I snuck in two of those feet\nto lift the bottom of the cover a bit so it didn't jiggle.
\nSo yeah, it's good now. But things could fall apart, break, who knows. When it does, I'll patch it, but in the meantime I hope to make\nsome music!
\n![\"The](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2025/10/mandy-beast.jpg\")
\nNic Cage admiring his newly forged weapon
I'm forcing myself to show stuff off more. Usually I don't share until things are perfect, but this\ntime, I have a very imperfect work in progress.
\nAt the same time, however, I'm pretty impressed with how a couple of AI tools I'm using are coming together.
\nThe app I'm building is called "Notespad". I think of it like Windows Notepad where you're\njust working with text to jot stuff down, organize your thoughts, and then get it out to another\napplication...maybe a word processor or an email reply.
\nIn "Notespad", we're not working with text, we're working with music notes.
\nSame deal: get the notes in, work with them, organize your thoughts, and get them out to wherever\nhelps you most. Maybe a digital audio workstation like Ableton.
\nAs a starting workflow, I'm taking a song, isolating a section in time, and then extracting the bass part\nusing Facebook Demucs running locally in the browser.
\nAnd then I use Spotify's Basic Pitch to extract MIDI notes from that baseline.
\nFinally, I just copy and paste the output to a Sonic PI live loop!
\nAgain, this is just a workflow I wanted to nail to get the hard stuff out of the way. I do think\nthis might be the most valuable workflow I want to enable, but we'll see as time goes on.
\nHere's a video to show it all off!
\n\n", "date_published": "2025-02-02T00:00:00Z" } , { "id": "https://benfarrell.com/blog/2025-02-02-begrudgingly-vite/", "url": "https://benfarrell.com/blog/2025-02-02-begrudgingly-vite/", "title": "Begrudgingly Moving to Vite", "content_html": "Sigh.
\nDouble sigh.
\nIt was over a decade ago when I just wanted to give up on all the Javascript framework churn.\nI was decently happy with Angular (first version!), and having just become comfortable there after blowing\nthrough what feels like a dozen others, I wasn't eager to switch to the latest craze (at the time): React.
\nI never did, and have only used React a few times in favor of Web Components.
\nAs much as I like to say I'm not a React fan, a major factor in me continually snubbing it was the front-end\ntooling.
\nReact and Webpack seemed joined at the hip most of React's early days. Even though I stayed clear of React, Webpack\nwas pretty inevitable for a lot of projects on my various teams.
\nI've been through my share of tooling over my years...whether it's ANT, Make, Gulp, Closure, Grunt, whatever...it\nwas all fairly interesting to me. Webpack was just another more modern one, and I was eager to dig in.
\nUnfortunately, I realllllly started hating it.
\nWhat I hate about Webpack is all the complexity.
\nUltimately though, this complexity really can't be blamed squarely on Webpack. It was the state of the JS ecosystem we found ourselves in at the time.\nWebpack is an everything bundler. It'll handle your JS, Typescript, CSS, JSX, image assets, workers....really, whatever you\nthrow at it.
\nAnd it really needed to. To be a tool that hopes to provide value in the new JS ecosystem, it needed\nto do it all. Whatever your source code is and whatever needed to happen to it to make it to your\nbrowser.
\nMaybe you want to use Typescript, or Coffeescript, or new JS features that may not be in the browser you're\ntargeting, your code needs to be transpiled.
\nESM imports as a standard weren't a thing yet either, so even just concatenation of a bunch of source files to keep organized and sane was a need.\nLet alone all the different ways of importing popping up: CommonJS, ESM, IIFE...
\nAnd then your images need to be copied over to where your Javascript got bundled and dumped to (with the right paths of course).
\nThere's more. I'm sure I could go on all day, but the point is: Webpack needed to handle a lot. And it did!\nI could be incorrectly remembering, but I feel like at first it did an\namazing job. It just worked well with few, if any, changes to the configuration.
\nAs time went on, however, we demanded more and more of it. It delivered, but it was increasingly harder to\nhave everything configured correctly for everyone.
\nThe result was that if you needed to do anything that deviated from the norm in your project, you'd spend\nhours figuring out why something wasn't working. My projects deviate from the norm...like a lot!\nI ended up becoming the go-to person to fix other people's Webpack issues not only because I had patience to solve\nit, but also because I'd been through so many other front-end tools, I understood what it was doing under the hood.
\nMe knowing how things worked under the hood was a major reason I was excited about Modern Web's "Web dev server".\nIt didn't do 99% of the things that Webpack did, but that was OK!
\nESM imports have been available in browsers for quite some time now. TSC (the Typescript compiler from Microsoft) transpiled to Javascript\nfantastically. And this is really all you need from a tool to do development on your local machine.\nFor production, bundling can be done as a separate concern with a tool like Rollup, which itself is a fairly easy and no-frills bundler (though it comes\nwith plenty of plugins to cater to any needs you have).
\nAll Web Dev Server needs to do is:
\nAs a result, during development, there's no bundling! Just run this and your transpiler (TSC probably), set them\nboth to watch for changes and you're golden!
\nIf you have any hiccups along the way - the Modern Web team made Web Dev Server work with Rollup plugins (within reason)!
\nFor example, if you NPM installed a package that ONLY offers CommonJS (those require('myimport.js) statements), you could inject\nthe CommonJS Rollup plugin into Web Dev Server. It would package as ESM on the fly, hot off the Node.js server, to consume in your\nweb app.
But, in 2025, this should be rare....right? Right?
\nIn 2020, Microsoft Edge was the last one to adopt import { mything } from 'mymodule.js, but we're still\nflooded with dependencies that use CommonJS, or worse, some weird mishmash of whatever.
Over the past few years, I've seen the Google Polymer/Web Components team really try to push for ESM consumable packages.\nAnd I think to some degree it's helped, just not enough.
\nWe're left with the occasional package that just doesn't work. Or more infuriatingly, ALMOST works, but\nsome piece is left broken for browser importing.
\nCase in point Tensorflow.js! I opened an issue in 2020, and there's been\nsome movement, but I think we're left with some Node.js code in there from a low-level dependency that breaks\nin the browser.
\nIn more robust (and magical) front-end tooling, this can usually be automatically polyfilled, or at least\nconfigured to be polfilled. But with the minimal tooling that Web Dev Server provides, this situation\nis very much in your face and requires you to use a Rollup plugin.
\nAt first (and for the last several years), I've been very much in favor of this. It's as if you are a chef,\nand you want to be in control of exactly what ingredients you're putting in your cooking. If something is making your process\nworse off, you can easily identify it and make it right because you're working with basic ingredients and know every detail\nof what goes into your cooking and how.
\nAlternatively, complex front-tooling means that the chef just dumps ingredients into the skillet and hopes for the\nbest. Full admission: I am this chef, for actual literal cooking anyway. And like the front-end tooling\nenvironment, it usually works out. But, also like front-end tooling it can make figuring out what went wrong\nreally hard if you want to improve or tweak your recipes. Additionally, if you are making something complicated,\nyou're at the whims of figuring out through the abstraction layer of your front-end tooling, which may or may not\nhave any resemblance to the reality of what's happening under the hood.
\nI suppose to complete my cooking metaphor for my last statement: Let's compare complex front-end tooling\nto dishes made from pre-packaged mixes. If there's an ingredient you don't like or isn't sitting well in your\nfull meal, then you're probably filtering it out of the mix by hand, or you're reading the instructions on the box\nto reason out how to change that mix to your liking.
\nIf you used fresh whole ingredients from the get-go, it might be a bit more work at times to figure out\nhow to make them work together, but you're learning and aware of how things work and come together all the while\nmaking the next time easier.
\nThis is all good. And it FEELS good when you solve your problems and your project suddenly lights up\nin your browser. But like I said, I've been doing this for five years. I've seen weird issue after weird issue\ncome up. And knowing how to fix a problem can usually mean diving into a dependencies code to figure\nout what solution you actually need to use. I feel like it's only easy for me 50% of the time, and the other\ntimes I have to be a detective.
\nI guess what it comes down to is the fact that I'm tired of dealing with it. I'm currently working on a project\nI'm excited about using a library. Unfortunately this library isn't browser importable\n(embeddable via a script tag, yes, but not ESM importable).
\nSo here I am just trying to get a fun part of my project working against my larger app and\nI hit this import roadblock. The idea of me spending the next 4 hours debugging how to use these libraries vs ignoring what's\nwrong and making it just work...well let's just say the fun part won out.
\nWith Vite gaining in popularity and being used by my colleagues at work, I was already using it.\nI know it handles these situations very well, and is also not a complex beast like Webpack.
\nAnd with that I switched my application to Vite!
\nBut... I immediately started having problems. It turns out you actually do need to learn a few things about\nVite before just going for it.
\nThe first issue I had was around transpiling my Typescript to Javascript, and really where those files live.\nModern Web's Web Dev server does not handle transpilation. I think that's great! In my mind, it should be\na separate concern.
\nUp until now, I've been just using the Typescript Compiler (tsc). Its run both for just building, but also during\ndev serving. It watches my Typescript files and when changed, I get Javascript files wherever I want them.
\nFor an actual end-user application (not a consumable library), my .js and .js.map files live side by\nside with my Typescript in my file structure.
The above scenario is probably possible in Vite, but it doesn't seem to be the default. Some experimentation leads\nme to think that you import a JS file in your code, since it's already there, Vite won't bother transpiling it.\nAnd then your on your own with the Typescript compiler as a separate step.
\nThat's fiiiiine, but given that Vite includes ESBuild for handling Typescript, we should probably not overcomplicate things\nby making it a separate step outside of Vite.
\nSo, first step to get this project "Vitified" was to delete all my transpiled .js and .js.map files.
One "rule" that's stuck with me for the past few years is to always import the .js file instead of the .ts, or even\ninstead of not using extension at all. This is how Web Dev Server works, as it's a server first and only handles\nother stuff if you need it. Since your browser only supports .js files and not .ts, that's what you need to do!
Of course with tooling, like Vite and others, it should handle whatever you throw at it and it should magically work!
\nThe down-side is that if your code imports .ts files or leaves it vague and extensionless, then you're locked into using some sort of\nfront-end tooling for your code. It'll never work with just a server. Now for an application that needs\nto be bundled for production, this is probably fine...unless of course you copy and paste your code somewhere else.
For a consumable package you publish to NPM, however, if you import .ts or extensionless files, you're\nforcing your consumers to use more complex front-end tooling just as you're doing to handle these imports.
All that said, as a rule, I try to adhere to the standard of using the .js extension when I'm importing regardless of\nwhat kind of project I'm on.
And this is where I run into the second part of my Vite import mystery. Despite deleting all of my transpiled\n.js files, nothing was working. Trying all sorts of vite.config.json options, what it came down to is my entry point.
Like many front-end tools geared for making web apps, your application entry point is really your HTML file. Inside, you'd\nneed to have a <script> tag where your src is the javascript entry point of your app.
<html>\n<head>\n <meta charset=\"utf-8\" />\n <title>My app</title>\n <script src=\"/src/index.ts\" type=\"module\"></script>\n</head>\n<body>\n<my-app></my-app>\n</body>\n</html>\nHere, however, even if the rest of your imports are .js files, this one (if you're using Typescript) needs to be a .ts\nfile. If referencing Javascript here and not Typescript, it would appear that nothing in your project\nis made to be Vite's responsibility. It does make sense. I imagine that behind the scenes Vite takes this\nentry point, checks all imports downstream and if Typescript, does the job of transpiling. On the other hand, if your entry point\nwas .js, then nothing downstream is Typescript and Vite assumes it doesn't need handling.
I did get a bit of a warning from a work colleague about the Vite cache. When using Vite, there's a\n.vite folder inside your node_modules folder.
![\"Vite](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2025/02/vite-cache.png\")
Basically Vite caches what goes through it's internal\nbundler and transpiler. My colleague claimed that, while he wasn't sure, there was some weirdness on his\nteam involving stale dependencies.
\nHis suggestion was to clear the cache every time the dev environment is fired up. So basically add a clean\ntask to your package.json scripts and run that before starting Vite, like this:
\"scripts\": {\n \"dev\": \"npm run clean:cache && vite --open\",\n \"clean:cache\": \"rimraf node_modules/.vite\",\n }I wanted to be a bit more optimistic, so I didn't take this advice at first, but I did start having\nsome weirdness with two of my packages I was working on when I used npm link so that I could\nwork on them in context of my application without publishing them yet.
I noticed that sometimes things weren't updating in my project when I made changes to my NPM linked local package.
\nThis could easily be a mistake on my part, but I can totally see Vite not noticing and updating\nit's cache correctly when an NPM installed package changes without running install again.
\nI had one last Vite mystery when I switched my project over. I didn't even know it was Vite for\nawhile, my application was just failing on some assets.
\nI was using Facebook's Demucs audio separation tool.\nI'll write about how I was using it later - but like any other AI solution that works locally in your\nbrowser, you need to download and load models.
\nAs they tend to be rather large files, it seems standard practice to compress them. When I was using\nWeb Dev Server previously, I was using fetch to grab them and feed these .gzip files to my worker code just fine.
When I switched to Vite, suddenly, the Demucs code acted like these models weren't loading correctly or even corrupted.\nI took a look at the size of the network request for these .gzip model files, and something was off.
They were much bigger (around double the size) of the files on my filesystem.
\nAnd there it is...the Vite server was uncompressing the files for me before they hit my app! My code, of course, expected\nto do that task itself, so the data just became unusable.
\nAccording to this github issue, it's not even something that can\nreally be fixed. Instead the workaround is just to rename your files to not use that .gzip extension so Vite\ndoesn't know what they are! I named all my files whatever.gzipped.
So, once I got my issues worked out and my project up and running under Vite, it was time to ask myself\nif it's worth it.
\nThere were a few hiccups and mysteries to solve along the way. I tried leaning on the docs pretty heavily, but\nnothing was coming up for my situations when I had these problems.
\nHowever, it seems that my real issue was that I just needed to make a couple small concessions when using\nfront-end tooling like Vite that I didn't have to worry about with minimal or no-tool scenarios such as what Web Dev server\nprovides.
\nOnce I accepted that there's a tiny bit of a learning curve and Vite does introduce a bit of an abstraction layer with it's own reasonable rules, I\nappreciated the fact that I don't have to worry about importing non-standard dependencies anymore while also\nnot working with the over-the-top complexity of other front-end tools.
\nAt the end of the day, it sure feels like a complicated tool like Webpack offers similar capabilities\nto Vite. Maybe I should try Webpack again after I've been gone all this time. Maybe I'll run into more\nmysteries down the road that make Vite feel like Webpack. But for now, Vite is pretty nice and a good compromise\nat the very least!
\nThat said, I think I'll still continue using Web Dev Server for the packages I publish to NPM.\nI want to make sure these are consumable anywhere and by coding for the browser instead of a front-end tool,\nI can mostly be sure I am.
\n", "date_published": "2025-02-02T00:00:00Z" } , { "id": "https://benfarrell.com/blog/2024-12-01-webgl-with-claudeai/", "url": "https://benfarrell.com/blog/2024-12-01-webgl-with-claudeai/", "title": "WebGL with Claude.ai", "content_html": "Back when I was making episodes of Web Components from Space, one of my last experiments was using\nAI to generate code in languages or environments that are just painful to work with.
\nWebGL is one of those environments. I'm jealous of my colleague who explored WebGPU and I mean to do the same in the future. It looks much better, although\nit STILL is experimental browser tech.
\nSo here I am trying to make stuff in WebGL.
\nIn my video series, I'd have AI write a shader for me when I tell it what kind of vibe I want to evoke. It barely worked, but it was\nfun trying to get it to work!
\nEither way, I'm keeping it up! I put a WebGL canvas in the footer of my website. Every so often when I write a new blog post, I'll\ndo my best to get the AI to cooperate and make a new shader for me.
\nI've had super bad luck with Github Copilot and ChatGPT in the past, so now I'm giving Claude.ai a go.
\nMy first prompt was:
\n"Can you write me a WebGL shader that appears really wide like a banner and evokes feelings of starting fresh?"
\nI had a bit of trouble with mismatched variable names across my shader code but ended up with 2 circles orbiting each other on the left side.\nHere's what I got:
\n\nIn my latest post, I wrote about playing a MIDI sequence with Tone.js.\nSo in my next Claude.ai prompt:
\n"please change this shader completely to invoke playing music or playing beats"
\nI ended up with a pulsing circle on the left side of my banner. I guess Claude REALLY likes circles. I thought it was too boring though,\nso tried again:
\n"can we try something that looks like it's visualizing frequency instead?"
\nUnfortunately, this time I got back non-working code. Something about arrays not being supported in WebGL
\nprecision mediump float;\n\nuniform float time;\nuniform vec2 resolution;\nuniform float[8] frequencies; // Array of frequency amplitudesWhen I complained it didn't work, and asked it to try again, I got back a something that looked like an EQ with frequency bars bouncing up and down.\nIt was pretty decent, but it was confined to the left side of the banner.
\nI did a couple more iterations before changing the code myself to make the bars span all the way across my banner, playing with size and number\nof frequency bars a bit.
\nUltimately, I'm happy - though it IS a bit ugly. I'll note that I do apply some CSS filters and opacity in my actual footer to make things a bit softer and subtler.\nAnyway, maybe I'll try suggesting colors in my next go-around.
\n\n![\"Claude.ai\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2024/12/claude.jpg\")
I've had a long-standing hobby that I haven't really leaned into since the mid 2000's. But all the same, it's inspired\nsome of my prototypes and experiments. And that's playing music. I played classical piano in grade school, then\nplayed in a few rock bands in high school and then after college.
\nJust these past couple years, I really got back into playing keys again, but also taking some vocal lessons. I think I'm decent\nenough that I need to start actually playing some songs. As I figure out what I want to play and what my sound is, I don't actually have a band...\nso how do I get a full sound? Since I'm huge nerd that writes software, now's the time to bring in some tech!
\nI won't get into what I'm trying just yet (and I'll note that the featured image is a new MIDI synth I'm trying) - but I AM finding that I need to work with MIDI a bit in kind of a niche way. So I'm building a small web app\nto work with short, one-track sequences of notes.
\nI spent lots of time building out the interactive timeline, which I'll talk about at a later date, but as I started wiring things up,\nplayback turned out unexpectedly hard!
\nTone.js is such a great project, and seemingly everyone's goto library for music experimentation since...well, since the Web Audio API first popped up\naround a decade ago!
\nI too have used it on more projects than I can remember, and it works well every time!
\nSo. What's the problem?
\nEvery time I've used Tone.js in the past, I'm playing notes live. They always get triggered in realtime by some aspect of an interactive\nexperience I'm making.
\nThis time, however, I'm importing MIDI and trying to play an entire sequence back. So, that should be easy! Let's just look over the\nTone.js documentation and see what we should use.
\nSo, even though the API docs are all there for Tone, suggested usage of these concepts for playing back a series of notes just don't seem to be found anywhere\nin the docs. Most of the how-to's seem geared towards simple playback of notes triggered in realtime, which is how I usually use Tone. And the rest\nis really just normal auto-generated API docs.
\nNot to docu-shame Tone.js, of course! It's a long-running project, and I'm come across messages from the maintainer saying they are\nonly one person with another full-time job, and they try their best! I'm sure over this project, there's been all sorts of features implemented for\nspecific use cases and I can certainly empathize with getting to write-ups for these "tomorrow" :)
\nOr you know - maybe I'm just not looking in the right place.
\nBut I did do a little trial, error, and experimentation to learn that I should probably just pretend I'm playing in realtime and manage my own sequence, which is NOT what I expected.
\nFor the use case of playing back a sequence of notes, it looks like I can count the Pattern object out right away. It seems good at what it does,\nwhich is to play a pattern of notes in a loop, but the API really looks like it's not meant for being more than a something like a simple\narpeggiator where you can't really alter the time or velocity or sustain....or really anything between notes!
Does that mean it's bad? No, of course not! Arpeggiators are super cool, but they just aren't the right construct to play MIDI notes\nwith all that nuance and subtlety.
\nHowever, when the Tone.js Pattern becomes part of your musical repertoire, you can do some cool things! For example, guitarland.com has\na great example of using a Pattern to play dual arpeggiators that, by using global delays, can be used to craft some interesting melodies that\nplay off of each other.
Sequences are another neat concept which ALSO doesn't quite cover my use case!\nThe Tone docs says that a Sequence is a series or array of events that happen at a regular interval, but with\nthe extra ability to have an inner array that subdivides the interval by the number of items in the inner array.\nThis seems perfect for arranging a melody with code and experimenting with timing without worrying about actual numbers, but with MIDI playback\nI want to allow explicit timing, allowing slightly off-beat notes and the like - and this Sequence pattern would make that hard, if not impossible\nto pull off!
I completely overlooked the Loop object until I did this post. Really, it seems to be something similar to setInterval where you run that Loop at a\ninterval you specify and do stuff in the callback. What's nifty here, is that you're using Tone.js so you can tell it to run at a timing relative to your beats per minute.\nThe following example from the Tone.js docs triggers every 8th note.
var loop = new Tone.Loop(function(time){\n\t//triggered every eighth note.\n\tconsole.log(time);\n}, \"8n\").start(0);\nTone.Transport.start();Given that this uses Tone.js and the browser's AudioContext to manage timing, it should be super accurate and reliable, unlike setInterval for example.\nHowever, that's all this really does! So any note sequences I'll need to manage myself. And to be fair, I do end up managing it all myself,\nbut I was REALLY looking for something in Tone.js to play a sequence!
After all my experimenting, I landed on the Part object...until I found out that it didn't work so amazingly for my edge cases.\nWas I right? Let me tell you what I found.
I used midi-json-parser to convert MIDI into JSON. There's not too much to say about that project, except\nit worked great! Ultimately, my MIDI events looked like this once I parsed and processed them:
\nexport interface NoteEvent {\n note: number;\n velocity: number;\n duration: number;\n time: number;\n}And then to play them back, I created a Part object like this:
\tPlaybackToneSeq.synth = new Tone.PolySynth(Tone.Synth).toDestination();\n\tPlaybackToneSeq.part = new Tone.Part((time, value) => {\n\t\tPlaybackToneSeq.synth?.triggerAttackRelease(\n\t\t\tvalue.note,\n\t\t\tvalue.duration, time,\n\t\t\tvalue.velocity / 127);\n\t}, events);So the above code is pretty easy to read, right? We're creating a PolySynth (or a synth that can play multiple notes at once) and a Part object\nwith our JSON parsed MIDI events.
And then when we want to start playback, we can just call start on the Part object and the Tone.js transport object:
\tTone.getTransport().start();\n\tPlaybackToneSeq.part.start();Once I got playback that seemed in time and what I expected, I was pretty happy after my confusing journey through the Tone.js API!
\nSo playback seems great, but I do want a fully interactive MIDI timeline. I'm going to be honest: initially I thought Tone.js was being\nbuggy and not so well thought out. But after revisiting to write this post and slowing down to think through the API, I think Part just\ndoesn't fit my use-case as well as I'd like.
The main problem is that there's no great way to query the current time of my playback session. Remember, my "playback session" is against\na timeline, so as a user I can scrub around, pause, play, and even define a range to loop over.
\nThe Part object DOES have a progress property, but it consistently returns 0 unless I enable looping and set the start and end times of the loop.\nHowever, when I do that, I can get query progress and get back a normalized time between 0 and 1 (or rather 0-100% of the loop).
Tone.js documentation tells me that I don't have to set the loop property to true. Instead I can set it to a number to indicate\nhow many times it loops. But in practice, I found that even if I set it to 1 to play a single time, that doesn't affect the main timing\nmechanism that I haven't mentioned yet, which is Tone.Transport.
The playback state of the Part will still tell me that playback has started, and there's nothing in the Part object that tells me it's finished,\nso I'd have to write my own logic to watch for that and take action. If I do nothing, the progress property will continue reporting time as though\nit's an indefinite loop, but no notes will be played once my loop count is over!
I say Tone.Transport, but the proper (non-deprecated) way to access it now is Tone.getTransport(). This transport object is the main timing mechanism\nbehind all of Tone.js. And it's global. So having a separate Part object when you have to ALSO go through the transport when you want to play, pause, seek, etc\nfeels a bit messy.
Given the above situation with Part and trying to get playback progress, then it seems best to perhaps just get timing info from the transport.\nBut like I said, when working with the transport, it's completely separate from the Part object, so it's on you to calculate time\nin relation to your sequence. For example, if you range select a portion to loop over, you'll need to adjust the global playback time\nyou get from the transport to the time in your sequence. Or if you leave the transport running while pausing your sequence (if you can), then\nyou'll need to subtract that time as well. Or, lastly, if you're on your third loop of a sequence, you'll need to adjust the time (as it's been counting up since the first loop started)\nto the relative time in your sequence.
To be honest, I gave up when I kept getting JS runtime errors thrown when I was\nscrubbing around. On my second pass with this code as I was writing this post, everything seemed to work fine. Maybe it was just\nme doing something wrong that first time around.
\nBeyond that, though, all of these problems are manageable assuming you're willing to write a bit of code to manage all of this.\nBut at that point, what is the use of the Part object? As I eluded to before, it's likely useful in other situations - I can\nimagine lots of use for Parts, Patterns, and Sequences in something like a music coding environment where you're creating a\ncomposition and want to experiment easily with different coding patterns.
That's not what we're doing here, though! So, ultimately, I think the best way to manage a MIDI timeline is to handle the sequence\nplayback yourself. I've had lots of past success with playing notes in Tone.js live, or triggered by code live, so that's exactly what\nI'll do.
\nBut how? Managing an array or note events is easy enough, but music is all about perfect timing!\nThe usual suspects are setInterval and requestAnimationFrame in Javascript, but I've been around the block long enough\nto know that these aren't great when timing is important. Both can easily fall behind when your UI thread is busy, and in fact that's\nwhat requestAnimationFrame is designed to do - to fire off when things aren't so busy.
And then the question is, even if these were perfect, what timing resolution do I plan to check for notes to play? If I was playing\na sequence where things aligned perfectly with a beat (quantized), then I could fire off notes according to 16th notes, 32 notes, or whatever\nthe needs are. But for MIDI, I want to allow for off-beat notes, and even imperfections in timing that make music sound more human.
\nIgnoring the resolution issue for a moment and focusing on accurate timing, I explored, or rather my co-worker found me, a small but nifty library called Zyklus.\nZyklus uses the Web Audio API to create a dead accurate clock to fire off events at EXACTLY the interval you specify.
\nI pulled this in locally and started playing around with it. It definitely delivers! I won't share much here because the demo page\ngoes into a lot of detail about how Zyklus works and lets you play with it.
\nInitially I was thrilled to see that it was a tiny ES6 package that could be simply imported into my project. Basically, I was just\nrelieved to see it wasn't commonjs, no need for front-end tooling, just a simple import!
\nUnfortunately, it was a little tricky getting it to work in Typescript. Zyklus uses the prototype object on AudioContext to extend out the\nnative audio functionality to include this new clock. I'm pretty sure we're still not cool with extending native objects like this (it was bad practice last I checked).\nAnd I was having a terrible time trying to get Typescript to play nice with it - so I just copied this small library locally into my project and added a big ol @ts-ignore to the top of the file.
// @ts-ignore\nAudioContext.prototype['createClock'] = function (...It seemed to work pretty well scheduling events! But then I realized something...I don't need to solve my off-beat notes. I don't need events to fire exactly when I need the notes to play.\nThe reason is that I can schedule Tone.js notes in the future!
\nconst now = Tone.now();\nthis.synth?.triggerAttackRelease(\n\tevent.note,\n\tevent.duration, now,\n\tevent.velocity / 127);In the above snippet, I'm telling the note to trigger at the exact current time of the timing event that's being triggered.
\nI changed this approach to instead schedule a note to play "now" plus the delta of the current time and the time of the next note in the sequence.\nIf it happens to be in the past, for whatever reason, then I just play it immediately (which is why I'm using Math.max).
const now = Tone.now();\nthis.synth?.triggerAttackRelease(\n\tgetNotation(event.note),\n\tevent.duration, now + Math.max(0, nextnote.time - this._currentTime),\n\tevent.velocity / 127);Going even further - since I'm measuring that delta as something that could be inconsistent, I really don't need to use Zyklus at all.\nBack to requestAnimationFrame for me! Assuming I plan the look-ahead time to be something big enough that RAF can keep up with, but small enough that I can stop playback on a dime, then I shouldn't\nnotice any playback issues.
When all is said and done, I have a Web Component and Lit based MIDI timeline player.\nObviously it's just a first pass, and I'll probably run into issues when I use this in a larger project, but for now it seems solid!
\nYou can find it on NPM here and the github here.
\nI thought this playback piece would be so easy, but there's obviously more nuance to Tone.js than I assumed. It'll be super interesting\nto see where my explorations go once I experiment more! Tone will be part of that, but Sonic PI should be another big part of it.\nEither way, I'll be making some music!
\n![\"My](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2024/12/synth.jpg\")
Whatever the heck this blog post is about, you can tell it's gonna be cool. Lasers are awesome.\nI remember I had a fifth grade field trip somewhere that had lasers, and I did a report on them because\nof how awesome they are.
\nFirst though, lets talk halftones.
\nBack around 2020, my extremely math-inclined co-worker Dmitry Baranovskiy had invented some algorithms for drawing halftones, as one does in their spare time :).\nHis experiments involved taking an image, and converting to a vector based halftone pattern. It worked super well, and he had a variety of programmatic settings to tweak them.
\n![\"sample](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2024/09/halftone-example.jpg\")
\nSample swirly circles halftone pattern alongside original image
We had different shapes, patterns like a spiral or wave for the halftones to follow, various levels of detail, all that. I don't really need to elaborate, because at the same time we teamed up and made a web app!\nWe launched "Hello Halftones" on labs.adobe.com.
\nAnyway, we launched our experiment Hello Halftones and OF COURSE we all had some fun with importing images and making halftone patterns.\nBut here's the thing - halftones are a bit of a niche thing. Sure they were super useful back in the day when newspapers relied on halftones to recreate images. But its now (checks watch) 2024\nand we can not only print super high res images, but also...things are digital! So halftones are a bit of an antiquated concept that get pulled out to achieve an\naesthetic effect every once in a while.
\n![\"Hello](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2024/09/hello-halftones-app.png\")
\nHello Halftones web app on labs.adobe.com
At the time, we talked on my team a bit about how it would be neat if we tried using these for laser cutting or engraving. But alas, we didn't have access to a laser cutter at work.
\nEdit: Unfortunately the makerspace I'm referencing is now unfortunately closed temporarily and under new leadership which I would describe as toxic
\nSo lasers were again calling to me from the recesses of my 5th grade mind in 2021 when we launched this project. But they were unanswered until\nI got involved with my local makerspace. But damn straight\nour makerspace had a laser cutter/engraver. Honestly, how could you not? Lasers are a cornerstone of the makerspace equipment repertoire.
\nNow, things never go as planned when you finally get to try the thing you said you'd try. Just because you imagined it going 100% to plan doesn't make it so.
\nAt the makerspace, we have a fairly powerful 100W Omtech laser cutter/engraver. Laser cutting or engraving definitely has a bit of a learning curve. And yes, part of that learning curve\nis reading the manual, which I never do. However, I made an exception here, BECAUSE. LASERS. ARE. DANGEROUS.
\n![\"Laser](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2024/09/lasercutter.png\")
We at the makerspace documented a lot of materials and the settings to use for each but\nevery project is different, and you just need to get a feel for what you're trying to do and allow mistakes.
\nMistakes are certainly what happened my first time cutting a halftone pattern!
\nOur board president Carla brought her dog Chewbacca (what a wookie) to the gallery on the day I needed a subject for my first attempt.
\n![\"Chewbacca](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2024/09/chewbacca-the-dog.jpg\")
Carla doing her best to get Chewbacca to sit still for a halftone portrait
\nAs I brought Chewie into "Hello Halftone", I noticed a couple of things right off the bat.
\nFirst, I had to go into Photoshop to both invert the colors of the image and adjust the brightness to make the halftone look good.\nThis certainly wouldn't be true for all images, but the lighting conditions and the background of the photo called for needing those\nadjustments for this to even work.
\nAgain, not all images need these adjustments, but I've found that lots do. And it was a little annoying going to photoshop all the time to do this, especially not\nhaving a live preview of the halftone right there as you do it.
\n![\"Chewie](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2024/09/chewie-in-halftone.png\")
\nNotice the dark background which we'd rather wash out, and the dark areas on Chewie's face we might want to raise up or down with brightness
Anyway, once I had a halftone I deemed good enough, I exported the SVG and brought it into Lightburn. For any of you that aren't familiar\nwith Lightburn, it's basically THE goto software for laser cutting/engraving. It's a bit pricey, but the alternatives are missing key features to allow you the ease of creating a job and the freedom to move elements around as you need. Plus its cross platform...so me being on a Mac isn't a problem.
\nThe halftone looked good, but the cut became kind of a mess. The problem was that some of the halftone shapes were too close to one another. At best, it created a brittle honeycomb like\npattern in the wood that fell apart when you looked at it wrong. At worst, large sections of the wood, those darkest areas in the image were collapsing into itself and breaking off the image.
\n![\"Portrait](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2024/09/laser-chewie.jpg\")
\nHalftone portrait of Chewbacca the dog, cut on our laser cutter/engraver
![\"Close-up](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2024/09/close-up-cavein.jpg\")
\nBrittle cave-in when halftone shapes are cut too close together due to large dark regions
I tried more and different pictures - and this problematic...let's call it a "honey-comb cave-in"...issue could be\nsomewhat mitigated by increasing the brightness of the image and trying to get large dark contiguous areas not to be so\ndark anymore, but for a web app that was all about dropping any old image in and getting a working halftone....well, let's just\nsay it irked me.
\nOK, no, I didn't actually call my app update "Laser Edition". But I did quietly add some features 3 plus years after we launched and deployed\nHello Halftones to labs. The first two things I added were, if you know web dev, just canvas filters in our implementation.
\nThe first was a brightness filter. So, in the web app...instead of going out to Photoshop, you can adjust brightness of your imported image.\nTypically, I've found you want to increase it - you're washing out all those huge dark areas and preventing them from being atomic\nhalftone shapes that are too close to one another.
\nThe second was a switch to invert the colors in the image. Again, this wasn't always necessary, but you never really know, given an imported\nimage, how your subject will mesh with your background. For photos with a discernible subject, you want the background to be non-existent, but sometimes the\nbackground is the most prominent thing there once it gets halftoned. So inverting colors is key!
\nThese features really do help, but sometimes you get a composition that looks really good, but the halftone shapes are STILL too close to one another\nand will definitely lead to HONEYCOMB LASER COLLAPSE (I've named this here hoping that I can make a good AI generated image with this prompt).
\n![\"Honeycomb](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2024/09/honeycomb-laser-collapse.jpg\")
\nHoneycomb laser collapse: a dramatic re-imagining generated with Adobe Firefly text to image
So, I added a third (and last feature) called "Max shape radius". Think of it like this: every atomic shape in a halftone pattern has a size (and therefore a radius).\nThe darker the color in the image, the larger this shape will be and probably a little to close to its neighbors. On a digital image, there's really nothing wrong when these shapes overlap or slightly touch each other.
\n![\"a](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2024/09/dark-halftone.png\")
Halftones generated from a darker section of an image with make the halftone shapes larger and closer together
\nBut when you're cutting holes with a laser cutter, as fine as the cutting beam is...well. HONEYCOMB LASER COLLAPSE.
\nSo this new "Max shape radius" slider says - "Hey, if pure black gets represented by a shape that's 100% of the possible size, let's limit\nthe possible size to some percentage of that maximum size".
\nI don't necessarily know if this is the perfect solution because as you go lower and lower, the sizes of your atomic halftone shapes start getting pretty\nuniform and you can't make out an image anymore. But, at the same time, it can serve to limit those darker patches to something that won't ruin your cut if you're careful.
\n![\"Max](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2024/09/max-shape.jpg\")
\nMax shape radius starting at 100% (left), then 84%, 71%, and finally 50%. Notice how the shapes successfully get separated in preparation for laser cutting, but the image gets washed out\nwhen going too low
In practice, I've also noticed that what looks a bit unsaturated on screen in terms of shapes being smaller and more uniform, actually ends up looking pretty great when cut. So there's a bit of a learning\ncurve and some eyeballing to do, but at least there is some control to constrain how much the laser cuts away.
\nAs I got the pieces in place, I was fairly set on the use-case of creating portraits of people. It seemed the best thing to shoot for, because\nportraits tend to have a clear subject and a background you don't typically care about, so it's something we can aim to wash out as we make halftone patterns.
\nBut even at medium halftone resolution, the finer details DO get washed out and the face looks less and less recognizable. And, I'm assuming if we go too fine, we risk\nthe honeycomb collapse again, or even just not having a pattern that lets enough light through to illuminate the image.
\nSo, with this in mind, I went for famous musicians! Portraits of legends that would be easily recognizable at a distance. It's laser cutting time!
\n![\"Bono](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2024/09/bono.png\")
\nSVG of Bono from U2, ready to be laser cut
Like I said, there was still a bit of a learning curve as I started out. I even got a BIT too ambitious once I had some great cuts where I wanted to try mixing acrylic ink with a clear\nsealant to fill the laser cut holes with a semi-transparent layer to diffuse light better.
\nUnfortunately, the holes were too tiny, and as a result air kept getting trapped making things super uneven and messy. That said, I don't typically work with these materials, so there\ncould be a better way of accomplishing this...though for now, I'm throwing in the towel!
\n![\"Filling](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2024/09/hole-filling.jpg\")
\nFailed attempt at adding light diffusing semi-transparent sealer - Notice especially on Bono how uneven it is on each hole due to air bubbles
Once I gave up....err...WAS SATISFIED with the results, I grabbed some cherry red wood stain I had in my garage and applied a few coats to the laser cut face. And then to create the rest of the box, used my\nhandsaw to cut the remaining pieces, nailing them into place.
\nI'm gonna be perfectly honest. I'm not a woodworker. My cuts weren't straight. Some of my nails went in crooked, and I really shouldn't have been besmirching the nice laser cut box face with nails, let alone crooked ones.\nBut as a first attempt, the results didn't look half bad.
\nCuttable LEDs strips are a thing! I didn't really know how they worked, but it was a fairly simple thing to research. I bought a 16.4 ft roll and some solderless/snap down LED strip connectors.\nSince I knew I was using low voltage, I felt safe enough to just touch the exposed wires to a 9V battery. After some trial and error figuring out the connection, my short 6 inch cut LEDs strip lit up!
\n![\"Powering](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2024/09/snap-led-connector.jpg\")
\nWith cuttable LEDs you snip along the marked area and expose the contacts. The snapping LED connectors can be a bit hard to open up at first, but once you do they hinge open and you set the LED contacts on the strip to the now exposed power connectors. Close the strip connector, and then touch the two separate wires\nto a 9V battery
At that point I knew my lighting plan should work! So, I measured the inside of my boxes and applied my (sticky backed) LED strip around the sides of the box and bought some 9V battery holders. These ones actually have on/off switches so they proved to be\na great way to provide power without leaving the box always on.
\nI did have a moment of disappointment when I turned on the box. The LEDs really weren't shining through all that well. The problem was that the light was mounted on the sides. I knew they weren't going to be directionally correct, but I\nthought that enough light would shine through the front. Unfortunately, this wasn't the case, so I lined the back panel of the box with an aluminum foil wrapped piece of cardboard.
\nTHIS was enough to make the light shine through the front well enough and create a nice effect.
\n![\"Boxes\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2024/09/three-boxes.jpg\")
\nBono box with LEDs and tin foil back panel in place, alongside John Lennon and the Division Bell album cover (non-illuminated)
![\"Inside\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2024/09/inside-of-box.jpg\")
\nInside the Bono-box: LEDs attached to the side, makeshift tin foil back, and 9V battery container
Though, if we're being honest, the LEDs now lit up the front too well. You could actually make out individual LED lights from certain angles.\nI gotta tell you, this was ruining my vibe. I don't need an EVEN light, just something noisy enough to not force you to see\nwhat the inside looks like.
\nSo at this point, I went to the library and studied up on optical physics...
\nJust kidding. I balled up some aluminum foil, and stuck it to the back panel with some tape and hoped for the best.
\n![\"Tin](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2024/09/balled-up-tin-foil.jpg\")
\nThe optical physics of carefully balled up tin foil disperses the light
Still having hurt feelings about my semi-transparent sealer attempt, I also picked up some colored gel light filters. I cut a blue one to height, and a bit wider than the box front and just stuffed it in. It's holding tight as its snug\nagainst the LED strip, so works well!
\n![\"colored](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2024/09/gel-filter.jpg\")
\nGel light filter curved to cram in the front against the laser cut face. The LED light strip holds it in place
So at this point, I had something that worked well enough, but I had uneven cuts and badly nailed sections.
\n"If only I could cut wood with millimeter precision accurate straight lines"...he said with a straight face and full access to a laser cutter.
\nThere's a fantastic website called MakerCase that lets you input the dimensions of a box you want to make and lets you download an SVG of the faces of the box ready to be laser cut!\nI made a 5x5x2 inch box and cut it out of 1/8 inch birch plywood. The cuts were perfect for snug assembly, and I added the halftone pattern to a couple of the faces, ultimately making a Lightburn template of sorts.
\n![\"Bonobox](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2024/09/bonobox-closeup.jpg\")
\nIs it a lightbox? Or is it a BONOBOX?
![\"Bonobox](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2024/09/bonobox-angle.jpg\")
\nPhoto taken from an angle, it's even better than the real thing
![\"Bono](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2024/09/bono-lenon.jpg\")
\nBono and John Lennon hanging out by my record collection
![\"Full](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2024/09/full-set.jpg\")
\nBono x2, John Lennon, and Pink Floyd's Division Bell
This was a fun little experiment, and while I'm not necessarily handy with wood, lasers made it easy. If you have access to a laser cutter,\ntry making your own laser cut light box!\nHere's my Lightburn template for the Bono box.\nThe template is for 1/8" inch thick material, and I used birch plywood.
\nBut if you're not in the area, you can always try out Hello Halftones and make your own halftone patterns for your own laser cutter, or\nany number of other things you can think of!
\n", "date_published": "2024-09-19T00:00:00Z" } , { "id": "https://benfarrell.com/blog/2024-09-04-long-time-no-blog/", "url": "https://benfarrell.com/blog/2024-09-04-long-time-no-blog/", "title": "Hi, Long time, no blog!", "content_html": "Well, well. Look who's back. Me!
\nMy last real blog post was in 2020, but really I hadn't been blogging much since 2019. And even that wasn't much. Last I was in\nprime form was in 2017.
\nHere's the deal. I get fixated on big projects, and I think it's to my detriment. I was happily doing OH SO MUCH blogging in the earlier\nto mid parts of the 2010's. They were all small little experiments, and I was learning and sharing a lot. But, often times\nthe feeling of "Yeah, I got this, lets move on to the next thing" starts creeping in. Let's go bigger, better, and splashier!
\nBlogging led to some great things! Lots of speaking, of course, but then in 2019 (I think) was when I officially started writing my book Web Components in Action.\nAs tech books go, I think it was fairly successful. As far as I know, I'm still the author of the only book on Web Components (with the fantastic Web Component Essentials being the one exception) and I've sold around 3,000 copies.
\nFrom there I felt like a gosh darn native web evangelist and it was my duty to spread the word. I felt like writing an actual print book with a publisher was a bit restrictive, and\nI wanted to get weird with it. So, I created a video series called Web Components in Space where I'd talk about Web Components and other web technologies in a spacelab. I'd interview people, do show and tells, and even do some experiments.\nIt was a lot of fun, and I got to interview some cool folks. I wanted to build on my past experiences with video, 3D, doing a podcast with my pal Adrian Pomilio, just intertwine a bunch of past stuff I did that I had fun doing.
\nIt WAS fun. But OMG, it was a lot of work. Between this and the book, my experimentation crawled and I was doing all production/writing stuff that took more time than I wanted.\nI kinda resented not playing with technology enough anymore.
\nAnd most recently, I went local! My wife and I moved to Hayward, CA in 2021. We bought a house, and especially after the pandemic when we\nwere online to a disgusting degree, I wanted to see REAL PEOPLE in my new REAL town. So, I joined the board of directors for a local non-profit. The Sun Gallery\nwas just coming off of winning a grant for a brand new makerspace.
\nThe other aspect of blogging was pushing it out somewhere. 10 years ago, Twitter was awesome! Now, what I can I say...Elon...happened I guess?\nTwitter just felt no fun anymore, and this was after I got sick of the whole web development community sniping at each other, especially for talking Web Components.\nAnd I'll be damned if freaking LinkedIn is the best place to be to connect over professional stuff, but that seems to be where we were.
\nBut honestly, back when Twitter was fun, it was kinda awesome to be blogging the longer form stuff and pushing it out on Twitter. It was a good time.
\nThe other thing is that I just got plain bored talking about Web Components ALL THE TIME. Yes, I've been using them for over a decade.\nI gave talks, wrote my book, and like I said, I kinda felt like I had this role pushing them.
\nHonestly though, I like exploring. Not telling people what to do. Just saying "Hey, here's some ideas, and here's what I did - try it and have fun!"\nWe're well beyond that with Web Components. They're an official thing now. They're not going anywhere. I don't need to be the guy pushing them anymore.\nI'll use them happily and all I ever wanted was to get to a place where I have a good set of tools to build with (that aren't React LOL).
\nI have that, and incidentally I held off using React for so long that I just officially had my first React project (in 2024!) and I disliked it\nas much as I expected. Though I do get why people were asking me about state with Web Components so much, though maybe that's just a consideration for function\n(non-class based) components. I was so horrified to learn that my constructor looking block (that obviously wasn't a constructor) kept getting called over and over again\nfor each render. Yo...can I just declare some normal class props please without declaring official state with an API call?
\nSo what changed? My wife asked for a blog redo on theGourmez.com. After almost 20 years of putting up with WordPress, I really didn't want to\ndo another WordPress site. I wanted to do something where I actually had control over actual real local files rather than a mySQL database where the content is locked up.\nEven though she had no idea what she was in for, nor how she was really going to write, I went Eleventy on her.
\nUltimately, we bought a Wordpress->Markdown editor, and I had her install VS code and Github desktop. She's been doing great with it.
\nBut yeah, just the notion of ditching Wordpress game me some thoughts about my own blog that I hadn't touched recently. Plus she's been on Bluesky, and that seems like a fine and actually pleasant\nmicroblogging platform to go to.
\nThe other piece of me getting back to blogging is that I've been doing (and meaning to do) some cool stuff lately. None of it is Web Components, though\nif I need to build something, OF COURSE that's what I'll use. I've been doing some generative AI experimentation like everyone else, but I've also been playing\nwith making digital art physical with the laser cutter and 3D printer at the Sun Gallery makerspace.
\nPart of my goal with writing will definitely be to do some marketing of sorts for the makerspace. But, of course it won't be phony advertising. I'm\nlooking forward to experimenting and playing as I always have been and sharing that to hype the place and hopefully attract some collaborators and volunteers.
\nMy other obsession lately is music. Back in the early 2000's I was in a band and we put out some albums, but I didn't keep up with music after we broke up\nand just became rather career focused. Though, if you look at my experiments over the years, lots were sound/music related.
\nBut in the past couple of years, I picked up playing keys again. However, I'm a sucker for alt/pop/indie rock and I need need need\nvocals to make it sound good to me. Never thinking I could sing, I stumbled into a great vocal teacher who's studio is a town over from mine in Castro Valley. I had low\nexpectations prior to meeting him, but daaaaaang this guy, Jim Bedford is a legend in the Bay Area.
\nI still have lots to learn, but I'm more than competent, and it's getting to where I want to focus on the whole picture. Initially I thought I'd just play\nkeys and sing, but then I remembered...I'm a huge nerd who builds weird experiments. So lately I've been experimenting with Sonic Pi, a Ruby based music\nprogramming environment, and to a lesser extent, SuperCollider, which is what it's based on.
\nThis is an exciting space, because I feel like a noob again, but I'm also an expert UX dev with a huge background in audio and video. This could lead to some pretty fun\nand hopefully rockin songs.
\nSo, I'm back and excited to get into this experimental, creative, fun, and just ALL OVER the place mode again. Blogging will hopefully be a great\nway to connect with people over my renewed interests, and also just a way to keep tabs on what I do. I always laugh at myself when I google how to do something\nand I end up on a blog post I wrote. See, even if nobody reads, it does serve a purpose!
\nFind me here and on Bluesky and feel free to say hi!
\n![\"whiteboard\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2024/09/newblog.jpg\")
\nyou can tell I'm serious because I made an experiments list on my whiteboard
I didn't do much hard digital work on this project. The Godzilla model was pulled from Thingiverse, and I\nbelieve it was this one.
\nAnyway, the motivation was that my wife wanted a Godzilla in her garden. I'm quite sure I didn't coin the term "Gardenzilla",\nbut that's obviously what it is, and we couldn't stop saying it.
\nInitially, I wanted to go a bit creative with it and saw an idea on Etsy where someone used it as a succulent\nplanter. I thought that was so cool, so I pulled the model into Blender and boolean operated out it's back.
\n![\"blender](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/portfolio/gardenzilla-blender2-smaller.png\")
\nthe Godzilla model in blender
![\"blender](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/portfolio/gardenzilla-blender-smaller.png\")
\nthe Godzilla model in blender from the back
When all was said and done, it looked great! My wife was happy I did this for her, but extremely skeptical that it could hold\ndirt and baby succulents, especially at its size. She turned it into an air planter instead.
\n![\"gardenzilla](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/portfolio/gardenzilla-planter.jpg\")
\nthe Gardenzilla air planter
But also - she REALLY wanted and envisioned a much larger Godzilla in her garden surrounded by plants.\nI have a Kobra Max 3D printer and it only goes so big, so I just created a bust from my model and scaled it\nup as big as I could possibly fit in the printer. This actually meant rotating the model at weird angle so everything\ndid fit at the largest size.
\n90 printing hours later, and one successful filament swap in between, we had a Gardenzilla! Of course, I had a lot of\nsupport material to remove because of the weird rotation angle, but it was worth it! I drilled a hole in the bottom and\nadded a garden stake to keep it in place.
\n![\"gardenzilla\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/portfolio/gardenzilla-raised.jpg\")
\nlarger Gardenzilla in the garden
As we at the Sun Gallery push our makerspace forward, this is one of the activities I researched and\nmade. I used our Omtech 100W laser cutter/engraver to cut the pieces. The design on the box are simple vector art assets I pulled from\nother things I drew previously. And the box, well that's pretty neat. You can customize the dimensions and create any size box at makercase.com.
\n![\"lightbox\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/portfolio/box-lamp.jpg\")
\nthe final result
![\"activity](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/portfolio/lampinstructions.png\")
\nthe activity guide I designed and created for the light box
One of the things I run at the Sun Gallery is a monthly jam session organized through meetup.com.\nIt's called "Jam & Toast", and the flyer/art seemed obvious, so I drew it up in Illustrator for our advertising needs.
\n![\"jam](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/portfolio/jamtoast.png\")
\nJam & Toast banner
As a board member of the Sun Gallery, we're always looking for unique ways to make art and promote\nthe gallery. Carla Dozier, the board president, had the idea to create a mascot for the gallery and drew up a rough sketch of Stan the Sun. I took that sketch and created a digital version of Stan, the Octopus.
\nOnce the digital version was complete, we used it in various places, but most importantly, Carla took my digital reference and\npainted a matching mural on the walls of the makerspace. I painted a bit, but my skills are all digital, and I was afraid I was screwing it up, so didn't do much!
\nLater, I went on to use a digital capture of the physical mural in designing some Stan stickers.
\n![\"sketch\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/portfolio/stan-sketch-01.png\")
\nthe original sketch by Carla Dozier (it's super compressed because I misplaced the original, and this was saved, downsampled inside Adobe Illustrator)
![\"final\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/portfolio/stan.png\")
\nthe final digital piece, leaving some room for text to be painted in the mural on the sign and scroll
![\"mural\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/portfolio/stanmural.jpg\")
\nThe mural as painted on the walls of the makerspace
![\"stickers\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/portfolio/stan-sticker.png\")
\nStan stickers! The tiny steampunk pirate ship is AI generated, but the rest is design/layout and a photo of the mural
![\"farmerstan\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/portfolio/farmerstan.png\")
\nA variation of the digital illustration to market a art and gardening class at the gallery
When we opened our brand new Makerspace at the Sun Gallery, I for sure wanted to make an exhibit!\nThis was when generative AI was just reaching the point where it was good, but could also be run locally.\nI took an older desktop PC I had, upgraded the graphics card and installed Stable Diffusion. I learned lots about running it\nthrough its REST API.
\nThe experience is created with Tensorflow.js body segmentation running live through a webcam, and the UI is done with Lit/Web components.\nThe code is freely available on GitHub.
\nIt's a fun little exhibit and it plays off the fears that AI will replace artists. So as you step up to the exhibit, you\npress a button and speak into the microphone to say what AI will replace you with, and then it generates the image, and cuts you out\nof the picture with body segmentation, and then replaces you with the AI generated image.
\n![\"sketch\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/portfolio/aiwillreplaceu.png\")
\nthe downloadable output after using the installation
\na video my wife recorded using the installation
\n", "date_published": "2023-08-15T00:00:00Z" } , { "id": "https://benfarrell.com/blog/2023-01-25-webcomponentsinspaceepisode5/", "url": "https://benfarrell.com/blog/2023-01-25-webcomponentsinspaceepisode5/", "title": "Web Components in Space Episode 5", "content_html": "This is a cross-post of the fifth episode of my spacelab based Web Components learning video series called "Web Components in Space"!
\n\nPlay this playlist for the intro, the run-down on what we'll learn and do in Episode 5, the show and tell,\nthe interview, and our experiment with SimplyRecipes.com to generate AI images from their accessibility text.\n
\nBen gives a quick run-down on what we'll learn and build today, the technologies we'll use, who we'll\nbe talking to, and what they built.\n
\n\nTools and technology we'll use today:\n
\n\n\nBen sits down with Christian Cantrell who describes his day-job as the intersection of creativity and AI. He's the VP of Product for Stability.ai. We chat about\nhow Stable Diffusion could be changing creative output, and delve into his writing process using image generation. We also talk a bit about Stability.ai and it's\nrole in offering APIs to empower all of this.\n
\n\nDue to unforeseen circumstances, Ben and his space lab come crashing back down to Earth. An effort is made to depict this with AI generated art\nusing Stable Diffusion. Once Ben is in the safety of his home office, he talks about running Stable Diffusion locally on an M1 based Macbook Pro,\nusing Node.js as a server to make the call into Python and Stable Diffusion, and finally replacing an image tag on a web page with a custom made Web\nComponent that loads the AI generated image. All of this is done with a hand rolled Chrome Extension. We'll cover it all!\n
\n\nIn this video, we turn to simplyrecipes.com and their recipe for orange breakfast rolls. We use our new Chrome Extension to swap images with those\ngenerated by Stable Diffusion using Simply Recipe's own accessibility text.\n
\nThis is a cross-post of the fourth episode of my spacelab based Web Components learning video series called "Web Components in Space"!
\n\nPlay this playlist for the intro, the run-down on what we'll learn and do in Episode 4, the show and tell,\nthe interview, and the five part lab.\n
\n\n\nBen gives a quick run-down on what we'll learn and build today, the technologies we'll use, who we'll\nbe talking to, and what they built.\n
\n\nTools and technology we'll use today:\n
\n\n\nBen shows off his already built music practice app called "BeSharp", available at besharp.app. More importantly, though,\nhe shows the audio technology that goes into it. Ben shows demos with Web MIDI, Tone.js, Pitch Detection with the Web Audio API, and a pretty great\npiano keys component. On the Lit side, he then teases how all of these technologies can come into a project using Lit's new Reactive Controllers\n
\n\nBen chats with Cassondra Roberts who is an all around amazing developer working at Adobe. However, you might have seen her around in the Web Component spec community, on the CSS working group, as well\nas helping maintain some prominent Web Component libraries. Cassondra came on to talk about her book in progress on Web Component CSS and styling. She also gives Ben some pointers for styling\nthe music app we're diving into today.\n
\n\nIn this video we explore Web MIDI. We'll build a quick MIDI keyboard connected demo and watch out for what note gets hit, whether its a press or release, and even how hard (or the velocity) of the press.\n
\n\nNow, we explore pitch detection using your normal browser connected microphone using the Web Audio API. We're heavily borrowing/stealing from Alex Ellis (https://alexanderell.is) here\nas his blog post "Detecting pitch with the Web Audio API and autocorrelation" does a fantastic job of accomplishing this\n
\n\nNext we try out Tone.js to make some noise. And how better to make some noise (if you don't have a MIDI keyboard) than\na nice little Piano Key Web Component?\n
\n\n
\nNow we get to take all of our audio demos and code, and make them into Lit Reactive Controllers to make them super easy to use from a Web Component app without\nlots of messy code intruding on your HTML markup!\n
\n\nWe're going to fast forward to having our demo work all functioning in an un-styled app. We'll explore some Web Component styling, especially exploring some of the advice\nof our guest Cassondra Roberts. We'll also talk a bit about using a mini design system to style the app.\n
\nThis is a cross-post of the third episode of my spacelab based Web Components learning video series called "Web Components in Space"!
\n\nPlay this playlist for the intro, the run-down on what we'll learn and do in Episode 3, the show and tell,\nthe interview, and the eight part lab.\n
\n\n\nBen gives a quick run-down on what we'll learn and build today, the technologies we'll use, who we'll\nbe talking to and what they built.\n
\n\nTools and technology we'll use today:\n
\n\nBen shows off some Tensorflow.js pose capture demos including bodypose, facemesh, and handpose before sitting down to chat with Google's Developer Evangelist. And then, of course, he gets caught up in animating\nhimself as an SVG boy with https://github.com/yemount/pose-animator.\n
\n\nBen chats with Tensorflow.js's lead developer advocate to find out what TFJS can do from an easy but incredibly useful hello world application, all the way to the cutting edge of what ML can do today.\nJason Mayes talks all about his early days on the WebML creating superpowers on the web as well as his lifelong goals of taking to the skies in every way possible.\n
\n\nLinks:\n
\n\n\nThis time we're using a template/starter project and not starting from an empty project. We try out both OpenWC's generator (https://open-wc.org/docs/development/generator/) and Lit's\nTypescript project template (https://github.com/lit/lit-element-starter-ts).\n
\n\nEach starter kit comes with a way to create docs, and so we go through a quick review of how either Storybook or 11ty docs are generated.\n
\n\nIt's time to start building out our first component to simply play video, and we do this by building off of and changing what the Lit template provides.\nWe also discuss building this component with Web Standard APIs only - specifically, not using Lit here because it doesn't make sense in the context of not\nthat much DOM or UI controls. But with that comes a little extra work that Lit would typically smooth over for us.\n
\n\nAfter taking the first stab at a video player, we tweak the Lit provided dev demo page to show our component off during development!\nNext, we organize some of our transpiled source. It gets sent to the root of the project and isn't intended to be published, so we edit our\n.gitignore file a bit to reflect that.\n
\n\nIn this step we create some playback controls. This is the one piece that we DO use Google's Lit for. It's helpful in terms of several\nstateful UI controls we have here. But, it's the only place we do use Lit, and these playback controls are actually optional for the larger\nvideo player component. These controls are enabled by the developer putting them into the video component tag as a child (or not). This is enabled\nby using "slots" in our video component.\n
\n\nAs we wire up events on these UI controls, the concept of subclassed events are introduced. These are much better than the old way of creating\ncustom events, in that the subclassed Event can contain custom logic and functionality right inside of the event, rather than logic spread all over\nthe place as we did before.\n
\n\nIt turns out that our video player can't scrub properly because of the lack of partial range requests on web-dev-server. This is fairly unique to our dev\nenvironment. And it turns out that we can install some middleware to handle this. We can grab this from a number of Koa.js packages listed on Koa's github page,\nand this is because web-dev-server is actually based on Koa.js\n
\n\nHere's the part that broke the Web Compon-o-tron! Tensorflow.js has some difficulties being imported as an ES Module, so we're going to try a Rollup bundling step of just those TFJS libraries\nto get it working in our project, and we'll have these bundled imports as source files that we can import easily.\n
\n\nIn this step we extend our object oriented hierarchy further. We build on top of video-element to create a videopose-element, and extend THAT out to create our specific pose solutions for hands, face, and body.\nWe get all of these Tensorflow models working with a live HTML5 video element as the source. This means realtime pose detection! We also normalize the points that are coming from these TFJS models\nsuch that they all follow a similar format. This will be important in the final steps!\n
\n\nIn this (almost) final step, we begin with all of our TFJS pose models working, but they only appear in the console. So now, we build out a visualization canvas to overlay over\nour video. This is another optional layer we can choose (or choose not) to add as a child element which occupies the slot of our component.\n
\n\nWe also try out the record function of our component and can download a full recording, with audio if preferred, of the pose we capture. Like where we started this step, just\nseeing data in the console, this download is just points. So we finish up by build another video-like player, but this player plays back just the captured points with an audio track\nif captured.\n
\n\nWith everything fully working, we could almost end here, but there are some final steps to make this ready to publish on NPM and consume in an application.\n
\n\nThis final step is basically just the main branch of this project, so, no github link for this specific step. There were only a couple of things to wrap up this set of\nTFJS pose detection components. First, we needed to make sure our Rollup bundled Tensorflow modules can be consumed from both the "src" folder for dev, and from the root of the project\nwhen using the component.\n
\n\nAnd then finally to close things out, we dive into creating documentation with the 11ty static site generator (https://www.11ty.dev), and I show off a simple demo app that\nconsumes our new set of motion capture components.\n
\n\nThis is a cross-post of the second episode of my spacelab based Web Components learning video series called "Web Components in Space"!
\n\nPlay this playlist for the intro, the run-down on what we'll learn and do in Episode 2, the show and tell,\nthe interview, and the seven part lab.\n
\n\nBen gives a quick run-down on what we'll learn and build today, the technologies we'll use, who we'll\nbe talking to and what they built.\n
\n\nTools and technology we'll use today:\n
\n\n\nBen shows off our special guest's component they built and maintain called "Shader Doodle" before sitting down to chat.\n
\n\nBen chats with special guest Merl. We talk about creative coding on the web,\nmotivations and inspiration behind Shader Doodle, how and if being a musician influences creating coding, and more.\n
\n\nLinks:\n
\n\nWe start our "Space Doodle" app with a bit of front-end tooling setup. For this project, we'll\nbe using Typescript and delve into using some web-dev server plugins.\n
\n\nTo begin, we start with installing 3 packages\n
\n\nNext, we'll set up some tasks in our package.json. New to us will be the Typescript compilation task, and\nthe task to transpile and watch our TS files. The serve task that launches our page for development has\nbeen covered in Episode 1 and adds TS transpilation to the mix. TS and serving are done with an ampersand\n(&) so they both execute in parallel. TS files will be watched and when changes happen will be transpiled to JS.\nAnd these JS files that were transpiled will force the page to be reloaded. We'll also need a simple tsconfig.json\nfile to kick us off with some light and relaxed settings for Typescript.\n
\nNow we're ready to create our app. We'll be creating our index.html file which, like Episode 1 will be styled and\nsized to the full page and doesn't scroll.\n
\n\nWe'll also add the doodle-app component/tag/element to our page body, and include a script link to our\n"Space Doodle" app entry point found at src/doodle.js\n
\nIn our application entrypoint, doodle.ts (which is Typescript), we'll create a mostly empty class. This class\nwill be our doodle-app web component which uses Lit by extending it. Inside this class will be the\nLit render call which renders nothing yet (by way of an empty html tagged template).\n
\nTo define the Web Component, we'll be using our first "decorator".\n
\n\nNext up, we'll demonstrate adding style to our component. Unlike Episode 1, we won't be using CSS in JS,\nor rather we won't LOOK like we are. We'll start by creating a doodle.css file and adding a simple :host\nrule to make our page red.\n
\nWe'll import this CSS in the same way, however it won't quite work yet. We'll need to create a web-dev-server.config.js\nfile to start editing our web-dev server config. Here we'll allow the CSS MIME type to be treated as JS, as well\nas using the rollup-plugin-lit-css Rollup plugin to wrap our CSS inside of Lit-ready JS, so we can import\nit into our component.\n
\nLastly, to make Typescript happy, we need to create a global rule to give any CSS we import a proper type definition.\n
\n\nWith all that, we have a full page application with simply a red background.\n
\n\nIt's time to add our UI. We'll be adding (mostly) non-functional UI as a first step of building our\napplication. We will NOT be paying attention to organization, so we'll be overloading our one single component\nwith all of our markup. For a real application this would be less than ideal - it's better to split things\nup more granularly as smaller and less complicated components, but thats not what we'll be focusing on today,\nso I'm allowing this project to get a bit messy.\n
\n\nWe start by adding Spectrum Web Components to our package.json. For Episode 2, I'm using dark mode in Spectrum\njust because in Episode 1 we used light mode.\n
\n\nThe application will be divided into two sections on the left and right. On the left,\nthere will be a canvas area where editable text overlays the shader-doodle component.\nBelow that there will be some controls for setting the text color and recording a GIF\nof the canvas.\n
\nOn the right, we have some dropdowns to allow us to load different sample shaders, and different textures (if applicable).\nAlso, one nuance of shader-doodle is additional configuration to run "ShaderToy" shaders, so there is a switch\nto turn that off and on.\n
\nBelow that is an accordion menu that will contain text editing capabilities for the vertex and fragment shaders\nfor the shader set. And lastly, a simple button to reload/refresh the shader after a user has made edits.\n
\n\nBut of course, none of this is wired up!\n
\n\nNow it's time to pop in our remaining components. Of COURSE we need Shader Doodle but, we'll need a code\neditor as well. I found something called lit-code which is a PrismJS + Lit based code editor.\n
\nWe'll add those packages to our package.json. Note that for Shader Doodle we're pulling the experimental alpha.\nWe should also import PrismJS so it's default languages like JS can have some nice syntax coloring in lit-code.\nIt might make more sense to set the language to HLSL (shader language), but it's a bit of a hassle to get\nworking, and I've found that it looks virtually identical to JS in practice (at least as far as the color styling goes).\n
\nStarting slowly by experimenting and adding shaders manually to our Shader Doodle HTML markup, we'll get a bit more\norganized and use the separate shaders.ts file to hold and export a set of sample shaders. We'll do the same with\nthe lit-code component. As a middle step, however, we'll store the entire shader script tag in a variable and use\nLit's unsafeHTML to render it, just to show this particular escape hatch exists when we need it.\n
\nLastly, we can add some dark style CSS vars to our 'lit-code' element to match the dark mode we already have for\nthe overall web app with Spectrum Web Component.\n
\n\nThe code editors and Shader Doodle will be a bit more difficult to hook up, but we can at least get\nthe easy stuff out of the way.\n
\n\nStarting with the color picker, we'll add a @property decorator. This is a reactive property, and is something\nwe explored in Episode #1. This time however, we can use a simple decorator instead of the wordy syntax in a\npure JS project.\n
\nBy hooking up the input event to the color picker, setting the current color attribute on each to this new textColor\nproperty, the color slider and area are now affected by user input. To finish up, we can add this CSS property as an\ninline style on the editable text element.\n
\nWe can do similar with the actual editable text content. For our purposes, however, a reactive property isn't needed\nhere. The only place this editable text will appear is here on this div. And then in a later step, we'll use this\ntext to render a GIF of our shader + text overlay. We just need to update this text property when our field updates.\n
\nSo after creating the property (non-reactive), we'll add an input event on this contenteditable div to simply\nkeep the property updated as the user edits it.\n
\nAlso easy, we can wire up our sliders that control how many frames we'd like to record for our GIF and time between snapshots.\nFor this, however, we won't use the @property decorator for these properties. Instead, lets use the more appropriate @state\ndecorator. This is new in Lit 2 and meant for internal properties that we don't care to expose outside of the component.\nIncidentally, it would have been more correct to use them for the colors and text. These reactive properties will now update\nthe label below the sliders to indicate what the timing of the recording will be.\n
\nTo finish up this left side, let's just fire off an alert as a placeholder for the function to record our GIF. So we'll\ncreate this placeholder function that uses the @click listener on the "Record and Save GIF" button.\n
\nLastly, we'll get the pickers/comboboxes working on the right side. These control the "shader" and optional\ntexture used for the shader. The menu items for the shader picker will be driven from the list of shaders found in shaders.ts\nwhile the textures will just be an array of images in our assets folder with the addition of a web cam and the option\nto not have a texture at all.\n
\nGiven that Shader Doodle is a bit abnormal of a component (I'll discuss why in the next step), we won't actually take action\non loading the shader quite yet.\n
\n\nBut wait! Even after we've wired this up, the picker menu is having some issues displaying when clicking to open it up.\nWe're back to the issue in Episode #1, where we get a process not defined error. Again, this is due to an overlay\nmanagement library trying to query if we're using Node.js or in a browser. We fixed this with a hack before, but now let's\nfix it properly.\n
\nIn our web-dev-server.config.js, we'll remove true from the nodeResolve object. This object is much like the lit-css\nplugin we're using, but nodeResolve is so important and central to how @web/dev-server works, it's a top level\nconfiguration option. Typically, you'll just want it turned on, so true is what you'd set this to. However, to give\nit more of a configuration, we can set this to an object. We'll do that, and use the exportConditions property to set this\nin production mode. Doing this injects the process object and allows the internal 3rd party library to know\nthat what we're doing is inside the browser.\n
\nSo here's the thing with Shader Doodle. It's pretty awesome, but doesn't seem designed to keep switching\nshaders on the same component instance. It seems designed to be setup once in code and then run. Our SpaceDoodle\napp probably falls outside the normal usecase!\n
\n\nThat's OK, we can work with this! Lit is designed to make minimal changes and not tear things down. This\nmeans that when an attribute or your component's slot changes, that's all that's going to change.\nYour component as a DOM element will not get torn down and re-rendered. Your constructor won't get called again\nand it won't get removed and re-added to the DOM.\n
\n\nBut in this case, this is actually the behavior we want! At least for this tiny section of our UI. We want to\nremove Shader Doodle from the DOM so that it gets torn down, and then add it back with a new shader such that\nit gets a fresh start. To do this, we'll fall back to the very behavior we use Lit to escape from: appending directly\nto the DOM.\n
\n\nSo, we're going to manually add Shader Doodle to <div id="shader-doodle-container"></div>. To do this, in our shaders.ts\nfile we'll be adding a createShaderHTML function which constructs a tag containing the correct vertex and fragment\nscript tags, as well as the appropriate texture custom element for the shader. But, we're going admittedly low-tech here.\nWe're creating this tag through string concatenation and then with the shaderUpdate method, we set the innerHTML\nof the Shader Doodle container to this string. But just prior tohat we set the innerHTML to an empty string.\nThis effectively removes the element from the DOM, and forces it to recreate itself in its entirety with the new\nHTML string.\n
\nIn this same shaderUpdate method, we'll update the code editors as well by using the setCode method of these lit-code\ncomponents. We'll also want the code editor components to update the shader. To do this we'll want to listen for the lit-code\n@update event. We'll create a brand new custom shader object with the new code by cloning the current shader object.\n
\nOf course to update all of these elements, we need references to them. And for this, we can use the query decorator.\n
\nLastly we'll use Lit's firstUpdated method to call this new shaderUpdate method against the first shader\nin the shader list.\n
\nLet's finally render a GIF from Shader Doodle. There are some technical challenges here, but first, let's\nstart with the UI.\n
\n\nWe're going to add another reactive property called recording. Even though it will be used internally\nin practice and can be decorated as @state, we're going to fall back to @property. The reason is that\nwe'd like to hide our recording settings when recording, but also show a progress bar when we do.\n
\nSo, first, we'll import the Spectrum Web Components sp-progress-bar and add it to our component markup.\nA @state decorated reactive property called frameRecording will also be added to our class. This property\nwill indicate which frame we are currently recording, and we'll use frameRecording vs framesToRecord to calculate\nprogress.\n
\nBecause recording happens for a limited time, and so will this state of our UI, we can set this new recording property\nto true or false manually to show either state. We also set it up to reflect so we can use the attribute in\nour dev tools to flip this property on or off and test the transition of the UI back and forth before we actually wire\nup the record function.\n
\nThe outcome here is that we'll be adding some CSS rules to show or hide elements depending on this recording property value.\n
\nAnd now the fun part, getting our GIF recorder to work. This is going to be a bit tricky, but we can show off some\nmore @web/dev-server plugins. Which, again, are actually Rollup plugins - so we have a huge library to choose from.\n
\nAs we install and import GIF.js, we quickly see that it won't work as is. First off, it's a Coffeescript based library, which even\nthe bundled version doesn't quite work as an import. So we'll reach into it's source folder, skip the Coffeescript and go straight\nto the JS files that originated as a Flash/Actionscript 3 library and was converted to JS over time.\n
\nUnfortunately even THESE JS files use CommonJS which don't work as an ES Import. So we'll fix this problem by installing\nand using a Rollup CommonJS plugin. This plugin will transform the require calls to be compatible with ES modules and allow us\nto use the library.\n
\nAfter this gets added to @web/dev-server, we find that there is an additional problem. Our browser starts erroring out because\nsome variables aren't defined before use. This problem is the result of this older library not conforming to strict-mode JS.\nES module imports actually do enforce strict mode, so there's no way around this problem.....or is there?\n
\nWe can use the Rollup Prepend plugin to insert Javascript code into this one problematic file as its served. We'll simply insert some\nvariable declarations at the beginning.\n
\n\nAnd with that, we can import our GIF encoder and set up the function and timer to capture frames, saving them\nas a GIF at the end of the recording period.\n
\n\nIn Episode #1, we never really finished our project to package it up for deployment somewhere. Rollup\nis a great choice for us, especially because we're already using Rollup in @web/dev-server for node module\nresolution and our various plugins.\n
\nSo, in this last step, we'll install Rollup and create a build task in our package.json.\n
\nWe'll start by npm install-ing a few packages. First, Rollup, and then some more plugins like @rollup/plugin-node-resolve\nto do the same module resolving work as our server, @web/rollup-plugin-html to manage our HTML page and linked script, as well as\nrollup-plugin-sourcemaps and rollup-plugin-clean to include sourcemaps and clean our build folder before builds.\n
\nOnce installed, we can create a Rollup config file and set it up in much the same way, using largely the same plugins\nand options as our web-dev-server.config.js.\n
\nLastly is adding our build task and running it. We can now run our bundled file with associate HTML file right in\na browser on any server, ready to be put online!\n
\nThis is a cross-post of the first episode of my spacelab based Web Components learning video series called "Web Components in Space"!
\n\nPlay this playlist for the intro, the run-down on what we'll learn and do in Episode 1, the show and tell,\nthe interview, and the seven part lab.\n
\n\nBen gives a quick run-down on what we'll learn and build today, the technologies we'll use, who we'll\nbe talking to and what they built.\n
\n\nTools and technology we'll use today:\n
\n\n\nBen shows off our special guest's halftone rendering experiments before sitting down to chat.\n
\n\nBen chats with special guest Dmitry Baranovskiy. Dmitry talks a bit about his life and early career\nimmigrating to Australia. But mostly we nerd out on math and making generative art/visualizations that led\nto halftones.\n
\n\nSo. Much. Nerding out. And then we throw in a little chess visualization just in case it wasn't nerdy enough\n
\n\nLinks:\n
\n\nSince this is the first episode and the first time we're setting up a lit application,\nlets focus on the first steps of laying out our application.\n
\n\nIn this first step, we cover\n
\n\nIn this second step, we're going to do some custom organization of our main app component,\nbut also build out our first child component that allows us to select an image to use for our halftone\n
\n\nStep 2 covers:\n
\n\nIn step 3, we're continuing our Spectrum Web Components journey. First we'll\nfill in the "floatingfooter" component with some download buttons.\nNext, we'll go crazy with the side panel and employ several different\nSpectrum Web Components\n
\n\nStep 3 covers:\n
\n\nIn step 4, we're going to stop hard-coding all of our attributes/properties and let\nthem flow through to our child components using Lit's "Reactive Properties".\nThe end result is that we can control all of our settings from the main component.\n
\n\nIn step 4 we:\n
\n\nIn step 5 we get our application working by finally applying values changed from our\nvarious UI pieces to our data model, which updates every piece of our application.\n
\n\nStep 5 covers:\n
\n\nStep 6 finishes up the application. We add a few final touches to make our application really\nsing.\n
\n\nStep 6 adds:\n
\n\nThis last step doesn't have a Git branch associated with it. We're just having a bit of extra\nfun as we play with our webcam, create an SVG logo for our application using what we've built,\nand then just recap what we've learned a bit.\n
\nIt's been a while since I've written a blog post. And even then it was a constant barrage of Web Components posts from me, leading up to a book I published with Manning called Web Components in Action. It's all because I believed in Web Components, but until recently (right around the time my book was released in September of last year), I knew they weren't quite ready, and I love to go on and on about new, rough around the edges technology.
\nThen something happened for me. Web Components became a bit boring. And I mean that in the best possible way. I use Web Components to build apps, and it's really just using the standard web toolkit in your browser (HTML/JS/CSS). I've adopted LitElement into my workflow, but even that isn't terribly exciting. There's a slight learning curve, but it helps a heck of a lot more than it hurts.
\nWhy is all of this boring technology good? Because I can focus on my REAL problems - those that exist in the projects that I'm building. I LOVE not giving a second thought to how to do something in a framework or front end tooling that has sixteen layers of abstractions that fight you at every turn.
\nAlso neat is rejecting complex systems (although when I say this as a 20 year software veteran, I feel a bit curmudgeonly). But now, it seems we have a new buzzword to describe this: "buildless".
\nAt its heart, "buildless" just means using an (almost) dead simple server for your dev workflow, relying on ES imports to load your Javascript (and maybe some other things). This all happens without any sort of bundling step - just code, save your changes and reload!
\nIt sounds great, right? It usually is, but there's some stuff to know that isn't so easy to look up in my experience.
\nI think it was LitElement and Spectrum Web Components that made me finally graduate to needing a bit more than a simple web server. As dependencies grow in a library you're using, especially if that library supports Typescript, the imports won't actually be pointing to Javascript files per se. Instead, best practices dictate that you leave the extension off such that your tooling can decide how to resolve the file. Moreso, "bare imports" are a thing we need to worry about for the same reasons.
\nTo use LitElement as an example, it's common to say:
\nimport { LitElement } from 'lit-element';
Where is the 'lit-element' package? Your JS doesn't know. And it's probably better off this way. It COULD come from your node_modules folder. You might want it to come from somewhere else depending on your project structure. Also, remember that these are relative paths. It could be 'lit-element.js', '../../lit-element.js', or '../../../../../lit-element.js' depending which folder you are importing from in your project. Having this as a "bare import" makes referencing something common like this headache free in your code.
\nOf course, that's just in your code. Something has to transform that path for your code to work. Ideally it would be something called "import maps", but these haven't landed in any browsers yet (except for behind a flag in Chrome). If you're making a bundle with Webpack, Rollup, or the like, those tools will transform your paths - but we're going "buildless", which means that something has to transform your path in real time when requested from the server.
\nAnd that's where web-dev-server comes in. If you happen to be familiar with "es-dev-server" already from OpenWC then it may be news to you that "es-dev-server" migrated to https://modern-web.dev. This would be, of course, on the realization that going buildless isn't just a Web Components thing, it's for everyone!
\nThe documentation for getting started with web-dev-server is pretty thorough for basic use cases. If you weren't familiar with going buildless, you are now. There's also the matter of "Well, how does the server know how to resolve the files?". If you're using an existing library that has ES module support with a proper ESM endpoint defined as it's main entry point, then it happens automatically when you use the node-resolve flag. There are more complex situations touched on here: here, but of course, it's not comprehensive.
So with that said, I'll tell you that the simple stuff, the stuff that's well documented isn't why I'm writing this blog post. It's when you have some more advanced use cases and need to use options you didn't quite know exist, or why they exist. Even worse is when an option that you need doesn't really exist in the standard web-dev-server.config.
Those are the situations that inspired this blog post. Like I hinted at the top of this post, I like technology choices that don't hide what's underneath. Unfortunately, to make something that's drop dead easy to use and install like web-dev-server, you DO have to hide a lot of complexity. And this makes it all the more difficult to piece together how you support something odd in your project.
\nModern Web did not pull a Scooby-Doo. There's no evil mastermind revealed when you pull off it's mask. Instead, the server is built on Koa.js. Honestly, I'd never heard of this project, but it looks to be an extremely well built and easily configurable server that runs on Node.js. On its surface it looks so good, I'm not sure if I should be embarrassed for not hearing about it earlier!
\nAnyway, with just the "web-dev-server.config", you'd really never know that Koa was lurking underneath as the heart of the server. You'd likely not care either, that is until you need middleware!
\nHere's what happened...
\nIn one of my projects, I was using the normal element. As part of the HTML5 Video API, you should be able to set the time by saying myVideoElement.currentTime = 10. Unfortunately, if your server doesn't support a feature called "partial range requests", video scrubbing doesn't work! Partial range requests are just an elaborate way of saying "I have a big file (like a video) - and I want to only get some bytes from the middle of this big file". Without this, your video file is downloaded in order, from the first bytes to the last.
Luckily Koa supports "middleware", and web-dev-server offers an easy "middlewares" field in the config file to pass through to Koa.
\n// the koa-range plugin is installed straight from NPM\n\nconst range = require('koa-range');\n\nmodule.exports = {\n port: 8080,\n watch: true,\n nodeResolve: true,\n http2: true,\n preserveSymlinks: true,\n moduleDirs: ['node_modules', 'bundles'],\n middlewares: [ range ]\n};\n\nIn addition to Koa middleware, your everyday web-dev-server.config supports a "plugin" field as well. This is well documented on modern-web.dev. Also well documented, but maybe a little surprising until you know it's there, is that plugins for Rollup may be supported in web-dev-server as well!
\nOne of the easier use cases is something I just came across as I was moving a project from Babel/Webpack to buildless. Despite future web standards that could allow us to import CSS and other types as ES modules, we have nothing now.
\nNormally, as I work with LitElement, I just wrap my CSS as something that can be consumed by LitElement.
\nimport {css} from "lit-element";\n\nexport const style = css`\n :host {\n height: 100%;\n width: 100%;\n display: inline-block;\n}`;\n\nHowever, this doesn't work so well if you didn't write the CSS yourself and are just importing from somewhere. I admit, there has been a rare moment I just wanted something to work, so I copied and pasted a small amount of CSS from my node_modules into a JS file.
\nInstead, we can use the amazing rollup-plugin-lit-css plugin wrapped in the Rollup adapter for using in our web-dev-server.
\nconst rollupLitCSS = require('rollup-plugin-lit-css');\nconst fromRollup = require('@web/dev-server-rollup').fromRollup;\n\nconst pluginLitCSS = fromRollup(rollupLitCSS);\n\nmodule.exports = {\n mimeTypes: {\n '**/*.css': 'js',\n },\n plugins: [\n pluginLitCSS({\n include: ['./src/**/*.css'],\n }),\n ],\n...\n\nObviously this is pretty specific to Web Components and LitElement, but all the same, it showcases a nice, simple adapted Rollup plugin. Also note the "mimeTypes" property - I stumbled on that the hard way, when my CSS files just weren't loading because they just weren't being accepted as files.
\nAnother very useful Rollup adapted plugin is @rollup/plugin-alias.
\nPicture this: You're happily churning away on a buildless project, but you come across a situation where you need some help from a project you found on NPM. You install it and try to import it in your project, but you get the error I've cursed at more than once "require is not defined".
\nBoooo. This is simply a package that cannot be consumed via ES modules. Or maybe, the package.json main field hides that fact that it DOES have some ESM goodies in there, instead pointing to the CommonJS entry point.
What do you do? Well, again, modern-web.dev has some good documentation on just that. In my work, however, I took a slightly different approach. Instead of doing it in realtime with the Rollup CommonJS plugin as they do, I use this same plugin to pre-bundle as CommonJS. So I end up with a "my-npm-library.bundle.js" file that I toss in a "lib" folder in my project's source.
\nNormally I can just stop there and import this JS. But this particular project that I'm doing at work, it was important to leave the bare imports intact. This means I don't want to import { some-import } from 'libs/my-npm-library.bundle.js', because I want this code to still work in other front end tooling situations such as Webpack (where I was moving away from).
That's why I need my import to stand as import { some-import } from 'my-npm-library'.
And THIS is where the Rollup Alias plugin comes into play:
\nconst fromRollup = require('@web/dev-server-rollup').fromRollup;\nconst Alias = require('@rollup/plugin-alias');\n\nconst pluginAlias = fromRollup(Alias);\n\nmodule.exports = {\n plugins: [\n pluginAlias({\n entries: [\n {\n find: 'my-npm-library',\n replacement: `${process.cwd()}/libs/my-npm-library.bundle.js`,\n },\n ],\n }),\n ],\n...\nSo you see here, it finds "my-npm-library" and replaces with the path (and file) I actually want. Notice the very Node.js specific process.cwd() piece of my string that just provides me with the path to where I'm running this server on (for me, the root of my project).
One piece of additional trouble with some of my bundled CommonJS files is that you have to go through a "default" object to get to the good stuff. For example:
\nimport { some_import } from 'my-npm-library' doesn't work because the Rollup bundling process produced an object that looks like Module.default.some-import. This leaves me with potentially writing code to say:
import * as MyLib from 'my-npm-library';\n\nMyLib.default.some_import\n\nSo this doesn't work for me, because for starters, it doesn't preserve my code as is for other tooling environments. But also, if you're using Typescript (especially external libraries), you've just changed the definition of how to consume this library - and Typescript has NO idea you'd need to go through "default" to get things done.
\nAnd this is why I end up creating adapter files for these situations. I can import my JS bundle, and make my exports work exactly like they did before.
\nimport * as MyLib from './my-npm-library.bundle.js';\n\nexport default MyLib.default;\nexport const some_import = MyLib.default.some_import;\n\nAs a brief sidebar, I'll say that I'm rather new to Typescript. I had a misconception that I could configure TS to bake my new resolved paths in when it transpiles. For me, it was an easy mistake to make given that I my project was previously in Webpack/Babel, and it was hard to know who was responsible for what.
\nWhen I started going down this path to buildless, I really did think that one of my options was not to worry about bare import resolving, or imports with no extensions. I figured Typescript would handle all that stuff when I transpiled (if only I could find the right options).
\nNope nope nope. Typescript is not designed to do that and there's been at least one issue on their Github repo where they've refused to do it!
\nAgain, maybe it's just me - but when you go down this path to buildless, it really helps to have clear boundaries of which technologies are responsible for which roles.
\nOne last thing I'll touch is the Rollup Node Resolve plugin. This plugin handles the core functionality I'm talking about throughout this post: resolving your modules, whether they be bare modules or extensionless files. By setting the nodeResolve flag to true in your config file, or simply specifying --node-resolve in your command to start web-dev-server, you're using it.
This simple flag covers most situations you'll come across. But of course, there are always exceptions. Enter my use case:
\nI'd been using Spectrum Web Components in my project, but just know that this will apply to any situation where you have an import in your main app's node_modules folder, but another thing in your node_modules folder has the same dependency and it reaches into it's OWN node_modules folder.
\nTo be more specific on my case. I was importing a Web Component. It imports sp-button to register a button component with the tag. This works great in my application, but this same application had a super complex component that I kept in a separate repo. That component ALSO imported sp-button from it's own dependencies.
We're now importing sp-button from "myapp/node_modules/@spectrum_web_components" AND "myapp/node_modules/mycomponent/node_modules/@spectrum_web_components". I was actually in a monorepo using PNPM, so these were the exact same files in my setup (just hidden under different aliases), but my application sees them coming from two different paths, and thinks they are different files.
This means that those side-effects (the stuff that happens when you load an import but don't necessarily call anything yet) happen twice, when they should only happen a single time.
\nIn my use-case, this meant my app broke because my Web Component was trying to register an already registered component tag and threw an error. What I needed was a way to make sure that Spectrum button only looks in my main node_modules folder (and because of that, the file only gets loaded and processed once).
\nAfter what I can only describe as an embarrassing amount of research, I saw that the Rollup Node Resolve plugin's "dedupe" option was the thing I need. But how are you supposed to use it?
\nThere's a couple of things that aren't obvious here in terms of the "nodeResolve" option in your web-dev-server.
\nFirst, that "nodeResolve" option is to activate Rollup Node Resolve as a plugin in web-dev-server. This top-level flag is essentially a shortcut (likely because it's a main use-case for the server) to adopting Node Resolve into your plugin list.
\nSecondly, in your config file, this "nodeResolve" param is a boolean OR an entire "nodeResolve" configuration! This is what I mean when I say that this level of (necessary) simplicity hides complexity in a negative way. To be clear, I wouldn't change this behavior, it just hides a bit in obscurity!
\nnodeResolve: {\n // Do not automatically resolve our bundles, instead use Alias\n // Regex here says to resolve everything automatically except the embedded strings\n resolveOnly: [\n /^(?!.*(my_npm_library|different_npm_library))/,\n ],\n\n // Resolve these in the root node_modules folder and not in child ones\n dedupe: [\n '@spectrum-web-components/button',\n...\n\nIn the above configuration snippet, we're configuring "nodeResolve" to "dedupe", or just use the top-level project node_modules only for the listed modules.
\nI should also mention, that depending on the load order in some situations I've had (as far as I can figure it), my "nodeResolve" was trying to handle those JS bundles I intended for my Alias plugin. So I've included some Regex here in the "resolveOnly" object to tell it not to handle the "my_npm_library" or another fictitious "different_npm_library" module in this plugin.
\nTo close out, I'll just say that this is by no means a comprehensive guide. In fact, not really a guide at all. This post is just meant to share my personal experiences with having to change out a large Rush/Webpack/Babel/Typescript project to drop the Babel and Webpack and go buildless.
\nUntil now, I've been using es-dev-server and now web-dev-server in a drop dead simple way, and it's been working magnificently! But as you've seen in this post, I found a handful of things that just weren't obvious as we get into more complicated territory.
\nEven though these are just some use-cases I've come across, I think the fundamental concepts behind them just weren't quite obvious for the average user, so I hope this post helps. In other words, even if you don't share my specific problems, maybe you'll have similar ones that can be solved in similar ways.
\nI'd like to thanks Ben Delarre and Westbrook Johnson for helping steer me in the right direction in a couple places in my exploration.
\nI should also add that this buildless choice works rather well for testing too! In fact you're largely using the same config options. And then of course, when you DO bundle for production, you can use those same config options further with a standard Rollup task.
\n", "date_published": "2020-11-23T00:00:00Z" } , { "id": "https://benfarrell.com/blog/2020-04-27-superheroes/", "url": "https://benfarrell.com/blog/2020-04-27-superheroes/", "title": "Superheroes", "content_html": "This was less of an installation, and more of a popup event! The de Young Museum in San Francisco was hosting a "de Youngsters Art Party" and Adobe was there to help out. We created a superhero themed installation where kids could draw their own superhero, and then see it come to life on a screen. The kids could then print out their superhero and take it home with them.
\nRead more de Youngsters Art Party
\n![\"the](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/portfolio/media_10fee2f7fc4d75641d11deea042d1ed94cc65cc2d.webp\")
\nthe popup setup
![\"drawing\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/portfolio/media_17b31d94526920c8e3611404aa820ac4cb47ba954.webp\")
\nkids drawing the costumes to be scanned in
![\"output\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/portfolio/83603322_10157035340999607_1368571825423908864_n.jpg\")
\nmy wife and I had our photos taken
![\"output\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/portfolio/media_1a8663338795cc6c139378e0c00b9deb0b2ef01bb.webp\")
\nthe printed output that kids could take away
I’ve been procrastinating on this post. It started out as an idea from my publisher (Manning) back in April, just as my book was getting closer to being wrapped up, to make a decent argument about why it’s a great time for the web developers of the world to start using Web Components.
\nI thought a post like this was a fantastic idea, so I immediately agreed. After all, a lot has happened in late 2018 and early 2019 to finally prove Web Components are ready to go.
\nThe reason for my procrastination was simple. At that time I’d started seeing some great beginner-oriented posts around Web Components that captured some of my same points and excitement. And, of course, as things always go on the internet, there were some Web Components killjoys that came out of the woodwork to stomp all over the excitement.
\nIt’s OK — no technology is for everyone or every use case. But as the internet goes, if someone prefers their technology over yours, they’ll definitely make sure you hear about it! I can’t think of a successful framework/technology/platform yet that hasn’t gone through this at least once.
\nSo, that’s where the Web Components discussion was in late spring and early summer of 2019. It’s why instead of this post, I decided to write a more technical one on the new Constructable Stylesheets feature in Chrome and how you can easily use a design system in your Web Components with it.
\nAnyway, I think people are finally a bit bored with arguing, and my book is now published (37% off with code: fccfarrell)! So, I think it’s a great time to revisit Web Components. Are they finally ready?
\nI think the easiest measure of whether Web Components are ready is whether they actually work. Can I drop one onto a page as a custom element and not have to worry about compatibility in modern browsers? Finally we’re in a situation where things are looking really great.
\nIn the past, the big holdup has always been browser adoption. We’ve spent far too long on the edge of our seats waiting for all major modern browsers to support Web Components, hoping the standard didn’t just fall apart as they sometimes do when browsers don’t actually end up supporting the feature you want. Prior to fall 2018, only Chrome and Safari were good to go.
\nIt all started to change in October, however. Firefox finally delivered Custom Elements and the Shadow DOM. While Custom Elements have always been an easy drop-in polyfill, the Shadow DOM is way more complicated and harder to use without native support. These two web standards are at the core of what we mean when we talk about Web Components. They enable us to create truly encapsulated bits of UI that can be wrapped up in a single tag and dropped onto a page easily without fear of conflicting with elements and style on the rest of your page, as we web developers have been plagued with for ages.
\nWhile Firefox was great news, it wasn’t the end of the story. Microsoft Edge was still the holdout. We knew that Web Components were under consideration and possibly in development, but that’s as much as we knew until recently. Then in December, Edge announced that they would be switching to Chromium, and Web Components would be picked up as a result. April brought us a developer preview release of the new Edge, and in the summer of 2019 we saw the same developer preview on MacOS, followed by the Edge Beta being released as recently as late August.
\nSo, now, we have Web Components support on all major modern browsers. As someone who’s been working with Web Components since around 2013, I couldn’t be more thrilled to see this vision come so far. Even better, we’re seeing some major usage. In February, Google’s Polymer team joined a conference call with other framework developers for a sort of “state of the union.” They reported that the Chrome team was seeing that 10% of all page visits in Chrome included Web Components usage. Of course, YouTube likely plays a big part of this, as well as other Google owned sites. Even so, we’re starting to see Web Components extend to other major sites like Github, and just now to Apple Music.
\nMission accomplished, right? In some ways, absolutely. The technical bits are there and people are using them successfully. It’s now time to redefine success and acknowledge that there are other aspects to this experiment that started several years ago.
\nWhen we talk about any new workflow or technology, even beyond web development, I would argue that there are three major aspects to discuss:
\nThere are so many nuances to each aspect, but optimistically, I think we can cross #1 off the list. Nothing works 100% for all use cases as “Custom Elements Everywhere” aims to demystify, but generally speaking, Web Components do work! Those naysayers that I mentioned above? They’ll argue over #2: the developer experience.
\nThe developer experience is so subjective that it’s impossible to gauge how good it is. It’s not just that developers have opinions on their favorite ways to code; it’s also what they build that varies so much. Because of this, talking about the right way to author on the web can get as heated as any political or religious debate. This is exactly where we were a few months ago: in a “back and forth loop” which had me procrastinating on this post.
\nLike I said, this lively battle is par for the course with anything. Any framework like React or Angular or Vue, no matter how good, has its haters. Zoom out a level and you’ll see the same discussions about the web in general. Or even Mac vs Linux vs Windows. To be honest, when people cared enough to begin passionately discussing their pros and cons this spring, it felt a bit like a rite of passage that took Web Components to the next level.
\nOK, so people are talking...and arguing. Does that mean Web Components are automatically amazing? Of course not! It does however surface some of the amazing efforts by the Web Components community, as well as some of the unmet needs that web developers have faced that Web Components might solve.
\nMy philosophy when writing Web Components in Action revolved around a big unmet need that I saw Web Components solving. There is a contingent of developers out there who love making stuff on the web but don’t want all the complicated tooling or a framework getting in their way. When I saw the “State of JS” results from 2017, I realized I definitely wasn’t alone. Even more recently, this post on “Going Buildless”caught my eye.
\n![\"\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2019/09/noframework.png\")
State of JS from 2017 indicating that “No Framework” was 2nd most popular behind React[/caption]
Web Components solve a huge need by making UI components a foundational part of the web. UI components have been around forever on other platforms. Components, at a minimum, provide your project with structure and organization. In my opinion, they started getting good on the web with Angular and React. But with those frameworks/libraries you get a lot of baggage. Baggage can be great if you use it, but if components are really the only thing you want, why bother with all of that other stuff in popular libraries when you get it free from the web platform? There was a tweet from a while ago that really stuck with me:
\n\n![\"\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2019/09/joshthomastweet.png\")
This was at a time when lots of folks were excited about Web Components and they made the bold claim that they would replace your favorite frontend framework. As you can imagine, there was lots of pushback on these claims.
\nThe tweet does a good job of setting expectations for those who are skeptical. It makes clear that Web Components, while providing a great foundation, do not provide features beyond making components first class citizens on the web. I’m thinking of things like routing, binding, and state management.
\nHere’s the thing, though: you don’t always need those features. I’d also add that in many cases, it can be pretty easy to write those features yourself, or pull in any number of tiny libraries that do that one thing well.
\nThat’s not to say that my philosophy is for everyone. Some of the recent debates are around simplicity and elegance in writing your code. Vanilla Web Components get you really far versus a normal vanilla JS project where things tend to devolve into code spaghetti. Code organization is so much better when you can group into components and nest them without worrying about style leakage. When you dive into the code for each component, however, things can get a bit verbose for more complex UI and you’re still left to your own devices for code organization even if it is on a smaller and much more manageable scale. Enter the “religious” wars where we’re fighting over the most concise and elegant way to write UI code!
\nWeb Components are starting to answer these calls in a big way, though. The obvious place to start is with the Polymer Project. The team is now focused on delivering LitElement and lit-html. To be honest, I was pretty happy with vanilla Web Components for so long that I didn’t give those tools a proper go until recently. I wrote a bit about them, mostly lit-html, in my book and discussed some things it was really good at, but until you’ve tried your hand at declarative programming, you tend to miss their true power.
\nDeclarative programming is really just a design pattern. It’s what React is known for, and now we’re seeing Flutter and SwiftUI adopting the same style. You’re essentially creating a data model and rendering a view based on that. With every change to the data model, you re-render, never touching the actual elements in the DOM. In the past, I’ve done things imperatively. The user clicks a button, then you manipulate the actual DOM element or elements to reflect what the user intention was. It can get messy, but with experience, it doesn’t have to be.
\nI can totally see why some folks coming from React are so adamant about adhering to this pattern—it’s really nice! But it’s really only feasible when the DOM can be diffed. Meaning: if one thing in the DOM changes, it’s going to hurt to re-render everything in the component or the page. You only want to alter the aspect of the DOM that changed from a render.
\nFor a while, React was the only major JS library to enable this. Enter lit-html. I’d argue that it’s barely a library. I’d call it more of a tiny imported utility that does exactly what I described React was doing. In lit-html’s case, however, the overhead from React is gone. Virtual DOMs are not created and diffed. The diffing is done on the real DOM. We’re left with an extremely lightweight, very targeted, and incredibly performant solution. As an ES6 module, it’s used easily with absolutely no frontend tooling or any extra help. Even React’s JSX is gone in favor of template literals and template tags native to JavaScript.
\nLitElement furthers lit-html’s declarative paradigm by creating a base class that extends your component to do some extra things that are normally verbose like keeping your properties and attributes in sync. With all the momentum behind LitElement right now, we’re seeing some great solutions on top of it. Those tiny solutions I mentioned before for vanilla components have a growing LitElement story as well. Do you fancy state machines? Try lit-robot. Want a great MobX solution for LitElement? Try lit-mobx from my great Web Component minded colleagues at Adobe.
\nDoes this mean that this is where Web Components are going? Not necessarily, it's one of the places Web Components are going. The Polymer Team is doing some amazing stuff, but Web Components aren’t one-size-fits-all. It’s a standard to build on top of, and lots of folks are doing just that.
\nLike hooks? Perhaps checkout the newer Haunted library, or it’s LitElement variation, which adopts the hooks design pattern popularized recently by React.
\nThere are a number of great projects that have sprung up using Web Components as their basis. Salesforce, for example, has an amazingly comprehensive component library and ecosystem. To be honest, I’m not knowledgeable at all on their wider development platform, but judging by their Lightning Web Components developer’s guide, they have something to be proud of.
\nIonic is another company embracing Web Components. Their Stencil toolchain actually compiles to Web Components. As a nod to the standard, and a bit of a shot at frameworks that tend to drift from it, they advertise that “Stencil doesn’t fight the web platform. It embraces it.” Ionic is fairly well known for their mobile tooling. With Ionic v4, they’ve switched over to Web Components with a nice library of components to go with it. Ionic Native should prove to be a great way to build Progressive Web Apps, mobile, Electron, and web from the exact same Web Components.
\nSpeaking of compiling to Web Components, a number of projects with a much smaller ecosystem than Ionic aim to compile standalone Web Components themselves. This includes projects created to compile React, Vue, and/or Angular components to Web Components. Another project worth mentioning lately is Svelte. Though the author, Rich Harris, has been very vocal about why he doesn’t like Web Components, Svelte offers a way to compile to them regardless.
\nVaadin Web Components were a very early entry into this space. While Ionic and Salesforce were building more of a platform/ecosystem, Vaadin (https://vaadin.com/components) stuck to providing a great set of components. I actually think it’s pretty interesting that one of their target audiences is Java developers. Java folks have plenty of concerns and choices of their own just like the frontend world, but I imagine that giving them UI components that just work takes a significant cognitive load off so they can focus on their Java ecosystem of choice. Even ignoring most of their UI components, the non-visual vaadin-router seems to be the component of choice for many Web Components developers who just need a router for their application.
\nI know I’m flat-out missing many entries into the Web Components space. In fact, most of the entries I’ve just mentioned, I haven’t even tried. It’s a lot to keep up with! It’s certainly not clear if there’s a horse to pick in this race. In fact, after I published this post, the author of webcomponents.dev chimed in hoping to be part of this round up! True overlook on my part, because its crossed my Twitter feed a few times. While I am currently set in my developer ways with my favorite IDE (hence not checking it out in depth), I can't argue the usefulness behind a Web Component ecosystem as they provide, complete with code editor and integration with a huge number of UI libraries. This also reminds me that I forgot to include Storybook as a way to build components in isolation, see how they interact together, and document them.
\nBottom line is that most of what I mentioned seems to be a “right tool for the job” situation. Underlying them all, though, is this new set of standards. It’s why in my book I decided to ignore most of the frameworks and libraries and simply focus on you, the browser, and plain HTML/JS/CSS. Grasping the basic concepts behind Web Components will give a great foundation for how any of these libraries work.
\nDo all of these great projects mean that you’ll never use plain HTML/JS/CSS in practice? Nope! I’ve hung on to writing Web Components this way for a while and it works well. As I’ve said, though, lately I’m really enjoying using lit-html for more complex UI, and perhaps I’ll dip my toes into LitElement soon as well. However, I’ve also seen and worked on some components that aren’t complex because of their UI but are complex because of other reasons.
\nOne such component is the Shader Doodle component that I lightly contributed to. The goal of this project is to componentize amazing WebGL shaders that work on the popular Shader Toy website so they can run anywhere simply by dropping the <shader-doodle> element on a page. This is a great example of a Web Component that has no UI. There is simply a canvas and some script tags. Really, there’s no reason to bring in even something as lightweight as lit-html here. The only element used here is a <canvas>!
![\"\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2019/06/sahderdoodle.png\")
A simple shader-doodle example as seen on Glitch
I’m also working on a few 3D and Mixed Reality-based components which I hope to share soon that also only use a <canvas>.
![\"\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2019/09/modelviewer.png\")
Tweet from the official
To me that’s the power behind Web Components. They are just a new set of standards that work for a huge number of use cases. And this is why in my book I chose to go as close to the browser and the standard as possible, empowering readers to get familiar with the foundations and continue on in any direction they choose.
\nNow that I’ve covered the tech and developer experience, from an admittedly non-comprehensive view, community is the last point to get to. It’s also arguably the weakest link when we ask ourselves “Why Web Components Now?” But it also presents a pretty big opportunity for engagement with a smaller community that is growing every day.
\nWeb Components in their current form and level of support are quite new. There is a wealth of articles dating back to 2011, but many are dangerously outdated. There were even a few books that sprung up in 2015, but these are unfortunately also outdated because the standards have been changing.
\nThat said, in the past year, and increasingly in the past few months, I’ve seen quite a few blog posts on Web Components come my way. Obligatory “to do” app included! Additionally, the first modern Web Components book was released last November by Corey Rylan. I’ve bought the book myself to compare notes, and, quite frankly, I think that both his and my book approach things just differently enough that both are worth a read and you’ll get different and valuable takes.
\nAs much as the community is really getting started, an argument can be made that now is the time to get involved. There’s lots of ground to cover! Web Components aren’t React or Vue or Angular. They are a standard that is reaching across many different use cases in many different ways, and as such, do need a lot of different voices chiming in. How can you add your voice? One way is to join the Polymer Slack channel. You might have mixed feelings about a corporate overlord like Google being the voice of Web Components. That’s perfectly fine, but the fact is that the Polymer team is really trying to help this community succeed however they can. Yes, you’ll hear more about LitElement and lit-html there, but the folks have been extremely nice and very willing to help each other.
\nFor more neutral ground, though it’s still Google-owned, webcomponents.org is another great Web Components resource. Because it’s already getting a bit dated, the Polymer team has put out a public call for ideas on rebooting.
\nDespite the Polymer team doing a fantastic job, there probably should be a resource that’s not owned by a company. Open-WC is exactly that. There are a number of great open source projects created by Open-WC authors to help the Web Components ecosystem. One big leg up for Web Component development provided by Open-WC is their es-dev-server. I begrudgingly got on board with their server vs a typical dumb http server. Sometimes I try to be a browser purist to my own detriment. ES6 module importing doesn’t need to be complicated with long paths to your node_modules folder. Import maps should provide the ultimate solution, but until they get wide support in all browsers, es-dev-server is a pretty nice thing to fire up and get developing!
\nThe big point to come back to, is that yes, the Web Components community isn’t as well formed as other dev communities that have been around for ages. We’re really just getting started, and there’s lots to do, and many voices to listen to. It’s an exciting time to get involved. If you’re in the Bay Area, you might want to check out the first Web Components meetup on September 23rd (unfortunately full up, but check the wait-list). Or, if not, keep your eyes on thisdot.co for more online meetups like this one.
\nAgain though, if you’re brand-new to Web Components and just don’t know where to start or where they will lead you, you might want to check out my brand-new book: Web Components in Action.
\n", "date_published": "2019-09-22T00:00:00Z" } , { "id": "https://benfarrell.com/blog/2019-06-03-adopt-a-design-system-inside-your-web-components-with-constructable-stylesheets/", "url": "https://benfarrell.com/blog/2019-06-03-adopt-a-design-system-inside-your-web-components-with-constructable-stylesheets/", "title": "Adopt a Design System inside your Web Components with Constructable Stylesheets", "content_html": "As someone who makes stuff on the web, there are two things that I've been seeing quite a bit lately: Web Component discussion and CSS debates. I think that Web Components, or more specifically the Shadow DOM, is poised to solve some long-standing CSS problems. I'm a big fan of Web Components. In fact, I'm just wrapping up a book with Manning Publications now, called Web Components in Action.
\nLet's quickly review where we are with CSS. Personally, I really dig working with CSS, but I never got super fancy with it. Whenever I start working with Less or SASS, or start adopting BEM or similar methodologies, I keep coming back to just writing plain, no-frills CSS. Under normal conditions, what I'm doing is not maintainable...like, at all. One article that popped up on my twitter feed recently is an argument against the Cascade. What?! "Cascading" is the first "C" in CSS!
\nSimon is right, though. Or, as right as you can be when generally speaking for all developers ever who make stuff on the web. Big projects have lots of CSS. As much as I love CSS, the more you have, the more brittle your page becomes. Rules start combining and snowballing together, until you're debugging some crazy hard to find style or layout problems. It can also become a bit of a game of Wack-A-Mole. You spend an hour figuring out why a rule broke the thing it did, change it, but that breaks something else that you thought was unrelated.
\nIt's no wonder solutions keep being invented to manage this mess, including the latest CSS-in-JS and CSS Modules (not the upcoming CSS Modules browser feature). These two lean pretty heavily on your JS skills, not to mention your front-end tooling setups. I'm not going to argue against any solution that tries to solve a nasty problem that we've had for as long as CSS has been a thing, but I will say that I wish things didn't have to be so complicated. I wish we could just use normal, straightforward CSS again.
\nThese days, I do! And it's thanks to Web Components and the Shadow DOM. The Shadow DOM is the metaphorical moat around your UI component castle. It keeps out invading armies of selectors (both CSS and JS querySelectors).
\n![\"\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2019/06/castleshadowdom.jpg\")
Saying the Shadow DOM keeps out selectors is an important distinction I've had to adjust to recently. I used to say it keeps out style, but something like the following actually does inject style through the Shadow DOM.
\nbody {\n color: red;\n}The above style globally affects everything on your page. As such, all text will now be red (unless overridden by a more specific selector). It's when you go deeper with some sort of selector, that the Shadow DOM successfully blocks your style. For example, if my Shadow DOM enabled Web Component contained a , we could style all buttons on the page leaving the Web Component buttons alone.
\nbutton {\n color: red;\n}The Shadow DOM doesn't let outsiders know what's inside. Your outside CSS has no idea that your Web Component contains a button, and therefore won't style it. The button selector has nothing to latch onto inside the Shadow DOM.
\nAnother way that styles can be let through is by using CSS Vars. These are simply variables that are defined in your CSS. If you really want that button inside your Shadow DOM to be red, you could define a button color var in CSS.
\n:root {\n --button-color: red;\n}Inside your component, your CSS could then use this variable to specify the button color.
\nbutton {\n color: var(--button-color);\n}All that is great - the Shadow DOM protects our Web Component from style intrusion, but how do you actually use CSS within the component? Well, it's not perfect yet. In my mind, perfection would be to just point to a CSS file and load it up, styling the mini-DOM of your Web Component. Instead, we're still relegated to using JS to do anything in the component.
\nAs with most elements, the shadowRoot property of your Shadow DOM based Web Component has an innerHTML property that you can set. You'll typically set this to a long string of HTML and CSS to represent an entire mini-DOM making up your component. Don't worry, it's really not as bad as it sounds. With template literals (`), and ES6 Modules to separate out markup into different files to not clutter up your component logic, it's pretty clean. I cover this approach very extensively in my book.
\nthis.shadowRoot.innerHTML = `\n\n :host {\n background-color: blue;\n }\n button {\n color: red;\n }\n #myspan {\n color: green;\n }\n\n\n\nExample HTML ContentExample Button
Regardless, we're still putting CSS in a JS file. It's not "CSS-in-JS", because we're not transforming it at all, but again, having a plain CSS file would be the dream. Aside from this minor hiccup, the brittleness in web development has been solved! Style won't infect our component from the outside, and style from our component won't affect the outside world. Notice in the code snippet above, where we're styling a button with no extra class specificity. This isn't just a simple example, it's fairly routine not to worry about doing something like this because only the buttons in this Web Component are styled this way. Similar with the span with an ID. You'd never use the ID attribute like this in a small UI component because the ID has to be unique to the page. Not so with the Shadow DOM, the ID only needs to be unique within the component.
\nUsing the Shadow DOM and Web Components is like going back to simpler times when web development wasn't so complex and fragile, because we've redefined the scope of a huge application or page, to a much smaller and manageable one. But, there is a major missing piece in all of this.
\nThe missing piece is a Design System, and that's the rub. We want to bubble wrap our component and protect it from all outside style, yet at the same time, we want just the right style to come in and make the contents of our component look like the rest of the application or page.
\nCSS Vars are just the about the only established way to do this, but doing things one variable at a time is a sisyphean task.
\n![\"\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2019/06/cssvars.jpg\")
Wedging a Design System into a Web Component today means likely exploding an established CSS system into pieces, turning the bits into Javascript strings, and figuring out a way to bring them all together in a meaningful way inside your component, only loading the bits you need. The other bad thing with this approach, is that you're creating a design system from scratch in each and every component instance on your page. Its tons of duplicated CSS inside every mini-DOM.
\nThere are two brand new browser features poised to solve this problem. The first is CSS Shadow Parts/Theme. After spending a little time experimenting with Shadow Parts, it became clear that there is a lot of work to do around changing existing CSS to use "part" attributes in addition to classes. The design system is just one piece of the puzzle. There's also a lot of onus on the Web Component developer to "export" parts through the the component into child components. The Shadow Theme feature sounds like it alleviates some of this, but unlike Shadow Parts, it's not even supported by Chrome yet while Shadow Parts are only supported in Chrome right now.
\nThe better option is the brand new "Constructable Stylesheets". It's not just better IMHO, it's pretty close to perfect, and I think is poised to bring us back to our basic CSS roots in the Web Component world. Not, only is it already available in Chrome, but is easy to polyfill as well.
\nConstructable Stylesheets are an evolution rather than a brand new feature. Really we're just extending the API of the Javascript CSSStyleSheet object. So, what's new?
It used to be that after creating a new stylesheet, you could only edit the list of CSS rules. Now, though, you can replace the entire sheet, wholesale.
\n\nconst sheet = new CSSStyleSheet();\nsheet.replace(`@import url('directory/cssfile.css')\n .then(sheet => {})\n .catch(err => {});Note that the above is using the async replace method. For loading stylesheets with the @import directive, the CSS won't be immediately loaded. That said, the new stylesheet is available right away.
The next question to answer is what can we do with that stylesheet? Well, now in Chrome, both the document and shadowRoot objects have an adoptedStyleSheets property. This property accepts an array of stylesheets.
So now, a CSS file, or multiple CSS files from a design system can be adopted by any number of Shadow DOM enabled Web Components on a page. Not only that, but these style sheets aren't copies - you're not loading your Web Component instances with tons of cloned design system instances as is the case today. Every component (and the document) can share the same sheets, as well as pick and choose which CSS to adopt.
\n![\"\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2019/06/adoptedstylesheets.jpg\")
I hope you're thinking this sounds as promising as I do! In theory, we can take a complete and unchanged design system and use it in Shadow DOM enabled Web Components! Instead of just writing a blog post that this is theoretically possible, I took that challenge on with a real design system. I just so happen to work as a prototyper at Adobe and love using Web Components in my work. Adobe's design system, Spectrum, is something I use almost every day. Of course, I haven't been able to use Spectrum in conjunction with the Shadow DOM, so I was really excited at the prospect of getting this to work.
\nSpectrum itself is pretty awesome, too. It's recently been reworked with CSS Vars as the basis of everything. And then, if a monolithic design system isn't what you're after, individual components are delivered as well. With Spectrum, a developer can layer on CSS Vars, the Spectrum base, the theme (light/dark variations), and finally a handpicked set of component CSS.
\n![\"\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2019/06/spectrum.jpg\")
No really, I don't just think Spectrum is awesome because I work at Adobe. It's awesome because this fits extremely nicely with Web Components and Constructable Stylesheets. Each component can use some simple JS logic to adopt exactly the CSS it needs. Every component adopts the base CSS Vars and base system style. We can choose which theme to use and load those files as well. Last, each component should know exactly which Spectrum UI components it uses, and also load those CSS files. This also means that the index.html page doesn't need to know anything about what components need to be included, nor link to any stylesheets itself. Every Web Component is completely self reliant.
\nAll that's missing is a global module that can keep a cache of all loaded sheets. Web Components can pull from this module, and if a CSS file has already been loaded, it will just deliver the cached sheet back. Before jumping in and getting Spectrum working inside my Web Components, I went to work and created Style Shelter (also available on NPM). In addition to caching, most sheets need to be adopted by the Web Components, but some (root level CSS Vars) need to be adopted by the document, so Style Shelter also handles adopting different sheets to different scopes.
I'm excited to say that my challenge to use Spectrum without changing any CSS worked like a charm! I knew I had to be thorough, too. Every CSS component needs to work, so I forked the Spectrum CSS repo and created a Web Component based demo page. I did run into some nuances to solve that were Spectrum specific, but you can read all about those details on the project's readme.
\nSo, browser support makes us come crashing back to planet earth. Right now only Chrome (and one would assume the new Chromium powered Edge) supports Constructable Stylesheets. Firefox and Safari supposedly are considering or are working on the feature now, however. Good news, though! There is a polyfill, and it's easy to use. The only downside is that styles are copied over and over again, just like I promised we didn't need to do. Take this Shadow DOM in Chrome, and notice that even though the component is styled perfectly, there's no style in shown - it's all adopted.
\n![\"\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2019/06/chrome-nopolyfill.png\")
Now, compare that to Firefox. With the polyfill, the component is styled the same, but we can see all the adopted styles copied to the Shadow DOM.
\n![\"\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2019/06/firefox-withpolyfill.png\")
So, hopefully Safari and Firefox deliver the goods reallllllll quick! Delivering an entire design system to a Shadow DOM with no changes is a really big deal. And I'm probably pushing my luck, but I'm going to need to ask all the browser vendors to deliver CSS Modules, too.
\nThe reason I want CSS Modules is not design system related. At the start of this article, I stated that I wanted plain, simple CSS again. Actual files, not CSS inside JS strings. I think it's incredibly important that a well-built and shareable component be self-contained and not dependent on anything in the outside world. You might guess we can use Constructable Stylesheets here too, but there's a small complication.
\nIn my Constructable Spectrum demo, I do just that. I load up each component's local style as an actual CSS file to be adopted. The problem is that stylesheet @imports are relative to the main index.html. So instead of pointing to ./mycomponent.css, I need to use the full path to my component's CSS from the root of the project. Not great. Web Components should not need to know where they live in a project to function. They should be able to be moved around and used anywhere without thinking about these things.
\nJS modules, however, are loaded relative to whoever imported them. CSS Modules should be the same, and theoretically, you'll get a CSSStyleSheet back...ready to be adopted. A nice bonus would be if the same CSS file is imported, it would be a reference to the same one that was loaded from a different Web Component. I don't know if that's the case in the spec, but it would certainly be AMAZING.
\nThe Constructable Stylesheet approach is just gaining steam now and only supported in Chrome. Because of it's uncertain future, I really couldn't put them in my Web Components in Action book. That said, I'm excited that approaches like what I've outlined are a natural extension of Web Components today.
\nWith the Shadow DOM, Web Components, Constructable Stylesheets, and possibly CSS Modules, we've got something great here. We're on the verge of getting simple and easy to use CSS back, and it's exciting!
\n", "date_published": "2019-06-03T00:00:00Z" } , { "id": "https://benfarrell.com/blog/2019-03-24-web-components-in-action/", "url": "https://benfarrell.com/blog/2019-03-24-web-components-in-action/", "title": "Web Components in Action", "content_html": "Due out in Summer 2019, Manning Publications is releasing my first book. Web Components in Action walks readers through the basics of creating their own custom elements, using template literals stored in modules to properly separate your HTML/CSS, and the ins and outs of the Shadow DOM. The book runs through lots of hands on examples, as well as build steps, testing and strategies to accommodate older browsers.
\n![\"\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2019/03/20041555/Farrell-WebC-MEAP-HI-239x300.jpg\")
During my time on the Design Lab team at Adobe, there wasn't one year that passed where one of my projects did not make it to the big after-hours stage.
\nMAX Sneaks is, in my opinion, one of the most exciting events at Adobe MAX. Prior to joining Adobe, I looked forward to watching all of the emerging technologies and projects that just haven't made it to the product stage yet. While working at Adobe, I've come to understand that it's not just the hard work of inventing the technology or workflow, it's carefully crafting a narrative around something for people to get excited about, and it's getting up on stage in front of thousands of people.
\nI personally haven't had the pleasure (or terror) of this, but my team members have, with projects I led development on.
\nMy first year at Adobe, we presented Project Louper, an image search feature with Adobe Stock in mind. It found similar images to the one you feed it, as well as weighted searches from multiple images. The designer I worked with, Bernard Kerr is up showing the project.
\n![\"\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2019/03/24004152/maxresdefault.jpg\")
The next year, my talented designer colleague, Danielle Morimoto showed off Project Quick Layout. It was a grid based layout solution where a user could chop up a blank page into grids super easy and populate with assets. The initial concept was designed and coded by another talented designer I still work with, Will Ruby. I took over development and made the concept more battle and production ready as we added more and more features. Erica Prenga did a kickass job as junior developer on this project.
\n![\"\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2019/03/24004645/99dc9b62-ed80-4713-b68b-df1ce594abda_rw_1920-1070x566.png\")
The next year, and the final year of working in the Adobe Design Lab, Yaniv De Ridder showed off Project SonicScape. Yaniv led development on this project and worked with myself and another talented designer, Michael Cragg. This Unity based project allowed a user to work with 360 video more easily, but more importantly, visualize and work with 360 sound alongside of it.
\n![\"\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2019/03/23232820/DMi5QzPVoAA_VfL-1024x603.jpg\")
This installation was a sound based one! We combined photos of the users of the installation with sound waves.\nThe sound waves were triggered by a MIDI keyboard and send to a speaker mounted near a thin layer of water. These visuals\nwould be combined to create the final output. We created this installation for the Museum of Photographic Arts in San Diego.
\nRead more Note to Self
\n![\"the](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/portfolio/media_16042252f528d1472a659e63f805f050555f2c9f6.webp\")
\nthe installation
![\"the](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/portfolio/media_1ae99bd62b5aec85ccc73fe4951253e86507e03ab.webp\")
the printed output
No, I'm not a "Machine Learning" developer, but I am having fun feeling out what it can do. All that's to say, this isn't an article about the gory technical details of Mozilla's DeepSpeech. Rather I'm writing my experience after spending a couple hours with it and being decently impressed and finding myself eagerly anticipating this project improving over time.
\nI've been peripherally aware of voice input tech for a while. Calling Google's Web Speech API "first" does a disservice to many others before it, but it was the first one I played with, and it's likely the first one many web developers like myself have used. It's also insanely good right now.
\nPart of why it's insanely good, is the fact that it can translate speech to text in essentially real time. Not only that, but to ensure results that make sense, it listens...but before it gives final results, it uses some natural language processing to figure out if what it heard actually makes sense...and then improves it. Using the Web Speech API, you as a developer can even see the results rated and scored.
\nGoogle is constantly improving. And it should. Their speech recognition is used on the web, on Android phones, and even on Amazon Echo competitor Google Home. They absolutely need it to be as perfect as possible so you, the user, can interact with experiences using only your voice.
\nOf course, Google isn't the only one in the game. IBM's Watson also does an exceptional job at this. Even better, their demo recognizes different speakers on the fly and labels them as such in the text back.
\n![\"watson\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2017/12/watson.png\")
Multiple speakers? An option to get word timings? Fantastic! Watson is positioned as a really good service for voice recognition for a variety of applications. Watson, of course, does tons of other things. It's actually used in "Star Trek Bridge Crew" to fill in some AI when you just want to play a mission and don't have a real life crew waiting in their VR headsets to play with you.
\nI'm also fairly confident that if I looked at Microsoft's Azure services I'd see the same, and in recent days you can see a similar cloud offering from Google
\nAs far as I'm concerned, these companies are doing good. Cloud services are obviously popular, and speech recognition that works is a great service. There's a problem, though.
\nEarly on, before Google had their paid cloud service in place, when their browser Chrome first started offering the Web Speech API, you could watch network traffic in your browser and see the endpoints they were using. For any application you wanted voice in that wasn't browser based - you could kinda sorta mock a service to their endpoint and shoot over chunks of audio data. It would do the same thing. I remember playing around with transcription of audio files via Node.js.
\nHonestly, this wasn't kosher. It was Google's service, and this is not what they intended it for. They even put a flag in their browser traffic to ensure it was coming from Chrome. Yes (sheepishly), I faked that too in my Node.js requests so I could continue playing.
\nAlso, check out this Watson pricing page. It's 2 cents per minute of audio uploaded. Yes, that seems super cheap. But it's 2017 and we're talking to our devices more than ever. Also, I have an idea for a project where I want to grab transcriptions for the entire Batman '66 run.
\n![\"MV5BMTkzNDY5NTg5MF5BMl5BanBnXkFtZTgwNzI4NzM1MjE@.V1_UY268_CR13,0,182,268_AL\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2017/12/batman.jpg\")
Yeah, the show only ran for 3 seasons, but it was on basically every single night of the week. It clocks in at 120 episodes of around 25 minutes a pop. That's 6000 minutes, or $60 for my stupid project idea assuming I don't make make mistakes. My stupid project idea might not even be all that stupid - I want to catalog and time speech. Video editors can spend a long time cataloging footage, or just searching for the right thing for the right cut. What if we could throw those 50 hours of footage at a speech and face recognizer overnight and have it ready for search in the morning?
\nPrice aside, there are data costs. Yes, I have unlimited internet at home, but what if I wanted to make a mobile application? Or a non or barely connected Raspberry Pi project? Voice is just one of those things that's becoming super necessary especially as we enter the new age of VR/AR. As inexpensive as Watson is with 2 cents per minute, its also potentially a bit cost prohibitive in large scale use cases.
\nThat's why I'm excited about Mozilla's DeepSpeech project. DeepSpeech is speech transcription service that runs locally using machine learning. The model they released is trained by way of Mozilla's Common Voice Project, essentially crowd sourcing the training for their model.
\nMozilla states that a Raspberry Pi and/or mobile isn't in the cards yet (unless you'd like to fork the open source project and figure it out yourself), but it is on their roadmap. I'm guessing that to make it more mobile ready, the model and associated data files will need to be cut down from the 2GB that it is.
\nI did have some trouble getting started, but I'll walk you through and show some results. Coming off of trying to get other ML libraries installed, this was a walk in the park and extremely straightforward. But, like i said, it's new and I had a bit of trouble.
\nFirst of all - I had Python 3 installed. Nope. Get yourself Python 2. It'll probably work someday on 3, but not today. Next their instructions to get started are super easy - run the Python package manager: PIP and do "pip install deepspeech".
\nUnfortunately, PIP couldn't find the package! Turns out Mozilla doesn't offer the package for Windows yet, and in fact looking over the docs Windows might not really be tested or supported at all. With my Mac at work, I figured I was out of luck - but then remembered that Windows 10 comes with Ubuntu now! Even though I was giving it a shot, I thought it'd be futile.
\nNope, worked like a charm! DeepSpeech installed quickly and easily. Next, I wanted to jump right in and give it a go. On their README, they list the command:
\ndeepspeech output_model.pb my_audio_file.wav alphabet.txt lm.binary trie\nThis begs the question of....Where are those files? That model file, the binary, the txt? Not at all obvious from the README, but you can find the on the Releases part of their repo.
\nOnce I had these in place, my first attempt threw an error. It was vague....something about having 2 dimensions.
\nTypeError: Array must have 1 dimensions. Given array has 2 dimensions\nAll it meant was that it doesn't support stereo WAV files, just mono ones. Somehow dimensions == tracks.
\nI used a YouTube downloader site to grab a few samples, then had them converted with FFMPEG. On a couple occasions, I used Adobe Audition to chop things shorter so things would only be a few seconds. You've got to be very careful here because your result can range from audio processing errors in your console or garbled nonsensical output!
\nSome tips:
\nMy very first try was a random example WAV file I found online:
\n\nIt was pretty good! The result was "A tree has given us a you net opportunity to defend freedom and were going to seize the moment and do it". The mistakes were "a tree" instead of "history" and "you net" instead of "unique". Honestly, I wonder if these methods would exist if we applied some Natural Language Processing as a filter like the cloud services do...and since we run it local, we can easily insert this step and many others. It took 10 seconds to process this 4 second audio file.
\nNow the real test, a two minute clip from Batman. Again, I ran this next video through a downloader service. It saved to WAV, but I had to run it through Audition to make sure the bit rate and sample rate were correct.
\nBatman talking to himself on Youtube
\nThe output was impressive, but there were long garbled stretches:
\n"o dascmissiur mister freeze wants what hello ill see if i can get a chief o here a moment commissoerdutchepfoherothisisbrusewiyistfreezeontswhatcommissionergardenisonlaealo with bat man mister wan and perhaps if we put the two force together and you could talk to him yourself all right chief i dont have much time oh that man yes mister wine i you heard mister freesesscirless demands just briefly if raidand i have is gobetweensare you prepared to make the telocacacminnightandpaytherensomisterwayei have no choice bad men that may i suggest you take the broadcaster the commissioners office in an hour earlier and we will have a dome package money a to me ackageyoumoney this sons risky risk is our business mister wine of course but an i have the same faginkyou that all of gottemcityaskihoperobinandi are deserving of that faith ill make the necessary arrangements a meetyouwithtaeconmster's office at eleven in it you can altakrmisuerindeed i did that man will usunup take telltecastanlavethedammypackageemoneywaitingsewateleventoliktohsofindmensodissimilaringbatyrisbenganyooensosimilaranoherh"
\nWhat's weird is that these garbled stretches look almost correct if they were spaced out.
\nSo yes, it has a little way to go, but it's impressive for launch. It'll will also only get better as Mozilla and the community improve the models, maybe create some NLP wrappers (or otherwise), and shrink it down for mobile. Congrats Mozilla, I'm impressed - this project is needed!
\n", "date_published": "2017-12-01T00:00:00Z" } , { "id": "https://benfarrell.com/blog/2017-09-24-the-slow-march-of-web-component-progress/", "url": "https://benfarrell.com/blog/2017-09-24-the-slow-march-of-web-component-progress/", "title": "The Slow March of Web Component Progress", "content_html": "Almost two years ago, I made a hefty series of posts on the promise of Web Components. Things have changed and promises were broken, but on the whole, I don't think MUCH has changed from an implementation perspective. These days, I've been sucked into the awesome world of the 3D web and WebVR and soon WebAR, but often I need some 2D UI around my 3D scene/canvas. When I do, it's STILL all Web Component based HTML, CSS, and pure vanilla Javascript.
\nYou'd think the biggest change might be version 1 of the Web Components API, but actually not much has changed from an implementation perspective. Really, some method names have changed, but the API is pretty much accomplishing the same thing.
\nHere's version 0:
\nclass MyCustomClass extends HTMLElement {\n // Fires when an instance was removed from the document.\n detachedCallback() {};\n\n // Fires when an attribute was added, removed, or updated.\n attributeChangedCallback(attr, oldVal, newVal) {};\n\n // Fires when an instance was inserted into the document.\n attachedCallback() {};\n\n // Fires when an instance of the element is created.\n createdCallback() {};\n}\n\nNow, compare that to version 1:
\nclass MyCustomClass extends HTMLElement {\n static get observedAttributes() { return [] }\n constructor() {\n super();\n }\n // Fires when an instance of the element is created.\n connectedCallback() {}\n\n // Fires when an instance was removed from the document.\n disconnectedCallback() {}\n\n // Fires when an attribute was added, removed, or updated.\n attributeChangedCallback(attributeName, oldValue, newValue, namespace) {}\n\n // Fires when an instance was inserted into the document.\n adoptedCallback(oldDocument, newDocument) {}\n}\nSo pay attention, here...what actually changed? The method names, for sure, but once you change the method names, the use is exactly the same. Bonus, we have a constructor! We didn't before, and its just plain nice to have something here to use as a callback when this component is first instantiated. Prior to this everything needed to be done when the element component is created or attached to the document. To be fair, component creation vs class instantiation seems essentially the same from a usage standpoint, but it WAS weird not being able to have a constructor on a class in version zero.
\nAnother small change is the observedAttributes getter. Previously in version zero, the attributeChangedCallback handler worked on any attribute of your component. Changing
static get observedAttributes() { return ['someattribute'] }\nSure, it's something extra to do, and yes, before I knew what this did, I spent several minutes trying to figure out why my attribute change method wasn't being called, but it's pretty minor and requires more deliberate intention. I can't really complain, the change seems good overall.
\nFrom a class implementation perspective, this is all that changed! There is one other change outside the class, though. It used to be that the class would be attached to the HTML tag like this:
\ndocument.registerElement('my-component', MyCustomClass)\nNow, in v1, it's done like this:
\ncustomElements.define('my-component', MyCustomClass);\nUnfortunately, while Chrome, Safari, and Opera support "customElements", Firefox and Edge do not yet. Given that Firefox is listed as "under development", and in Edge it's "under consideration", I'm OK with this. We'll get there, but in the meantime, a polyfill works.
\nOne of the biggest points of excitement for Web Components for me was the elegance of working with three separate things in combination to create a component: Javascript, CSS, and HTML. If you asked me 2 years ago what the biggest risk to this vision was, it was getting browsers to implement the Shadow DOM. To remind you, the Shadow DOM was a protective wall around your component. Components could have their own CSS associated with them, and this Shadow DOM protected CSS rules from the outside seeping in and wrecking your rules. Likewise, your components internal DOM couldn't be manipulated from the outside.
\nUnfortunately, browsers were slow to adopt this, and even worse, it was harder to polyfill. The Polymer project even invented this notion of a "Shady DOM". Given this confusion, churn, and uncertainty, I never really adopted using the Shadow DOM. In all honestly, I personally don't really need it. I can see bigger applications and teams using it as a layer of protection against themselves like how other languages might use private/protected/public variables in their classes as a way of allowing team members to use and call on only what's been exposed.
\nBut this is the web! When this layer of protection isn't offered to us, we just use conventions instead. Biggest and easiest convention is to just never tweak component DOM from the outside. If you need to do something like this, you're doing it wrong...just make a method as part of your component's API to do what you need.
\nCSS is a bit trickier, but we've had the tools we've needed since the earliest days of CSS. Instead of relying on the Shadow DOM to stem off outsiders from mucking with your component's style, simply namespace every single CSS rule relating to your component with the component's name like so:
\nmy-component .an-Inner-Class {\n background-color: black;\n}\nAll that said, it appears there is a new version of the Shadow DOM shaping up. I haven't followed the latest here at all, but I think I might wait until there's a strong indication things will settle down before I bother with it.
\nGiven than the Shadow DOM, for me, is so easy to ignore until I have more confidence, I'm not really bothered. What I AM bothered by is how "HTML Imports" have dropped from favor. To be fair, we've always been able to polyfill HTML Imports fairly easily. At the same time, though, when Webkit/Safari has no interest and Firefox has no plans to implement, the whole notion seems dead in the water. I've seen some conversation that the web community doesn't want to adopt HTML Imports in favor of the Javascript "import" mechanism, but I'm not aware that this works in a meaningful way yet for HTML, nor is "import" supported in any browser except the most recent version of Chrome and Safari.
\nThis leaves us with a bit of a challenge. I really don't want to create my component's DOM entirely with code - every single tag created with "document.createElement('div')" and then assigning classes, innerText, and then appending the child to a parent.
\nFortunately, I've found that for me at least, inlining HTML into my Javascript is not as bad as I thought it might be. Components themselves should be fairly small - if you want major complexity, you may want to architect your big component into smaller ones that work together. Therefore, the HTML that you inline shouldn't be that complicated either. By convention, I can also use the constructor for my component as a decent place to put my HTML, because there isn't much else I need to add here.
\n\n constructor() {\n super();\n this.template = '\n <h4>Objects\\<select class="fileselector">\\\n <option value="default">box primitive</option>\\\n </select>\\\n </h4>\\\n <ul></ul>';\n }\n\n connectedCallback() { this.innerHTML = this.template; }\nThe above component represents a simple list (ul tag) which has a header above containing some text and a file selection menu. Honestly, the example I pulled isn't the prettiest thing in the world right now, and once I flesh out this simple component, I expect to have double or triple the lines of HTML potentially. But, all the same, it's pretty manageable to inline this. It also introduces a couple simple things in the way I format my HTML. I properly indent and new-line everything here just like you would see it in an HTML document. The mechanism to accomplish this readability is simply with a backslash after every continuing line.
\nI've also been exposed to the concept of backticks: `. Backticks are another way to wrap your strings in Javascript that allow you to inject variables. This is more commonly known as "template literals". It's not a new concept by far. Though I haven't really done anything with string templating in the past, I believe the practice is extremely common in React, Backbone, and Underscore. I haven't favored the usage of this for HTML because I like to keep my markup and JS separate, but I think I'm caving now to get a decent flow for components.
\nOne problem with templated HTML in this case, though. It's easy enough to inject a var like so:
\n\n var inject = 'hi';\n var template = `<div>${inject}</div>`;\nThe problem is that in the simple example above, the "inject" variable is in the same scope as the template! Typically when I want to use this type of pattern, I prefer to store the template as a sort of variable I can access from elsewhere rather than having it inside my code logic when I'm constructing these elements.
\nHere's a fake example to explain:
\n\nfor (let c = 0; c < data.length; c++) {\n let myitem = document.createElement('li');\n myitem.innerHTML = `<div>${data[c]}</div>`;\n mylist.appendChild(myitem);\n}\nIn this example, I'm appending new list items (li elements) to a unordered list (ul element). Right inside my loop here, I'm declaring what my template looks like. Personally, I think this is some bad code smell! Ideally, I want to break out any HTML I have into a separate variable so that if I AM going to inline my HTML (which I still think is kinda smelly on its own), I should at least have it separated out so I can easily track it down and change it. Putting it inside my application logic especially inside a loop like this just feels terrible.
\nUnfortunately, it's not possible to save a template with literal like this as a variable. Instead, we can create a method that accommodates both this and the creation of the element:
\n\n itemTemplate(data) {\n var template = document.createElement('template');\n template.innerHTML = `<li class="mesh">${data}</li>`;\n return template.content.firstChild;\n }\nI use the "template" tag here so I don't have to decide upfront which type of tag to create, and my tag (including the outer tag) can live entirely in this template string. Otherwise, for my outer tag I'd also have to have additional JS calls to set any attributes, classes, or IDs on it.
\nCustom events haven't changed, but there's a little trick I like to use that's worth mentioning. Here's the creation and triggering of a custom event:
\n\n let ce = new CustomEvent('onCustomThing', { detail: { data: data }});\n this.dispatchEvent(ce);\nThe above code is pretty simple, but there is one thing I don't like about it, and that is the string 'onCustomThing'. If you think about it, whoever consumes this event outside this class needs to spell 'onCustomThing' correctly AND use the correct capitalization. If we change this over the course of our project, we could break things and not know it.
\nThat's why I like to assign a sort of a static constant to the web component class. In practice I haven't been using any JS language features that dictate it is a static constant (though I probably could copying how observedAttributes is declared). Here's how I do it:
\n\nMyComponent extends HTMLElement {\n ...\n disconnectedCallback() {}\n attributeChangedCallback(attributeName, oldValue, newValue, namespace) {}\n adoptedCallback(oldDocument, newDocument) {}\n}\nMyComponent.CUSTOM_THING = 'onCustomThing';\ncustomElements.define('my-component', MyComponent);\nSo now, elsewhere, I can listen for the event like so: mycomponent.addEventListener(MyComponent.CUSTOM_THING, e => this.onCustomThing(e)); Yesssssss, you could bungle the syntax here as well making it as bad as a string, but it's easier for an IDE to refactor and predictively type as you code.
\nThis last bullet point of what's missing is a minor one, and I think it's slowly being corrected. Web Components aside, I've been developing most of my projects using Javascript modules by way of the "import" command. Chrome's latest version supports it, though I haven't properly tried it out yet. Instead, I've been relying on the "browser-es-module-loader" polyfill. It works amazingly well and I use it as a way to give my application a class based "controller" that can import modules as it needs to.
\nSo you can import a "main entry point" Javascript file as a controller, and anything downstream can also import modules. It's pretty awesome, but any Web Components you use in your application are NOT downstream of this controller and as a result cannot use imports. I haven't put in any serious brainpower to overcome this, but instead when I run into this issue, I take it as a hint that my component could be getting a bit too complex, and I work around it. Honestly, though, once this polyfill is not needed anymore, I'll be happy!
\nAs a whole, I'm still happy with writing web components after 2 years. I still have no desire to change. I think things are getting better and better, just a bit more slowly than I originally anticipated. I'm also a bit surprised at HTML imports being on its last legs. As a workflow and architecture, I still think it holds up really well, even if we have to shuffle around some of the pieces that make it up.
\nEverybody is different, though, and there are many different tools for many different jobs. I still haven't touched React or Angular 2-4 yet. I'm happy, but if you use those frameworks, you might be perfectly happy too! Consider this another tool to add to your belt (without all the bells and whistles of course).
\n", "date_published": "2017-09-24T00:00:00Z" } , { "id": "https://benfarrell.com/blog/2017-07-14-a-bad-attitude-about-a-frame-i-was-wrong/", "url": "https://benfarrell.com/blog/2017-07-14-a-bad-attitude-about-a-frame-i-was-wrong/", "title": "A-Bad attitude about A-Frame (I was wrong)", "content_html": "I haven't written many posts lately, especially tech related. The reason why is that I've been chasing the mixed reality train and learning. 3D development in general has had a number of false starts with me, and I never went in the deep end until now. This past year or so, I've been using both Unity and WebVR.
\nPrior to this, I went through a Shockwave 3D phase in the early 2000's. People forgot about that REALLLLLLL quick. Likewise when I got all excited about Flash's CPU based 3D engine Papervision (not Macromedia made, but for Flash), I remember learning it and then the hype died down to 0 within a few months. And then of course, there was Adobe Flash's Stage3D. But as you might recall, that was about the time that Steve Jobs took the wind out of Flash's sails and it was knocked down several pegs in the public eyes.
\nWhatever your opinion on Director or Flash, it doesn't matter. Approachable 3D development never felt like it had a fair shot (sorry, OpenGL/C++ just isn't approachable to me or lots of people). In my opinion, there were two prime reasons for this. The first is that GPU's are really just now standard on everyone's machine. I remember having to worry about how spectacularly crappy things would look with CPU rendering as a fallback. Secondly, and this is huge: visual tooling.
\nDon't believe me? I think the huge success of Unity proves my point. Yes, C# is approachable, but also, being able to place objects in a 3D viewport and wire up behaviors via a visual property panel is huge. Maybe seasoned developers will eventually lay of this approach, but it introduces a learning curve that isn't a 50ft high rock face in front of you.
\nUnity is fantastic, for sure, but after being a Flash developer in 2010 and my career seemed to be crumbling around me, I'm not going to be super quick to sign up for the same situation with a different company.
\nSo, enter WebVR. It's Javascript and WebGL based, so I can continue being a web developer and using existing skills. It's also bleeding edge, so it's not like I have to worry about IE (hint: VR will never work on IE, though Edge's Mixed Reality support is now the only publicly released version of WebVR (FF and Chrome WebVR are in experimental builds)! Point being, is that all those new ES6 features I dig, I can use them freely without worrying about polyfills (I do polyfill for import, though....but that's another article for another time).
\nThose of us who were excited about WebVR early on, probably used Three.js with some extensions. As excitement picked up steam, folks started packaging up Three.js with polyfills to support everything from regular desktop mouse interaction, to Google Cardboard, to the Oculus Rift and Vive, all with the same experience with little effort from the developer.
\nI found that object oriented development with ES6 classes driving Three.js made a lot of sense. If you take a peek at any of the examples in Three.js, they are short, but the code is kind of an unorganized mess. This is certainly forgivable for small examples, but not for big efforts that I might want to try.
\nSo, I was pretty happy here for a while. Having a nice workflow that you ironed out doesn't necessarily make you the most receptive to even newer ways, especially those that are super early and rough around the edges.
\nI believe early last year (2016), when I was attending some meetups and conference sessions for WebVR and Mozilla made a splash with A-Frame. Showing such an early release of A-Frame was a double edged sword. On the plus side, Mozilla was showing leadership in the WebVR space and getting web devs and designers interested in the promises of approachable, tag based 3D and VR. The down side was that people like me who were ALREADY interested in WebVR and already had a decent approach for prototyping were shown an alpha quality release with a barely functional inspection and visual editing tool that didn't seem to offer anything better than the Three.js editor.
\nAs I wasn't excited about it at all, I also reasoned that the whole premise of A-Frame was silly. Why would a sane person find value in HTML tags for 3D elements?
\nMonths passed, and I was happy doing my own thing without A-Frame. I even made a little prototyping library based on existing WebVR polyfills with an ES6 class based approach for 3D object and lifecycle management. It was fairly lightweight, but it worked for a couple prototypes I was working on.
\nRight around when A-Frame released 0.4 or 0.5, the San Francisco HTML5 meetup group invited them on for another session in another WebVR event. The A-Frame community had grown. There were a crazy number of components that their community built because...hey A-Frame is extensible now (I didn't know that!). The A-Frame visual inspector/editor is now the really nice and accessible as a debug tool from any A-Frame scene as you develop it. Based on the community momentum alone, I knew I had to take a second look.
\nTo overcome my bad A-Frame attitude, I carved out a weekend to accomplish two goals:
\nI actually thought these might be unreasonable goals and I was just going to prove failure.
\nAs I mentioned briefly, I had my own library I was using for prototyping. Like I said it was basically a package of some polyfills that had already been created for WebVR with some nice ES6 class based organization around it.
\nI knew that A-Frame was built much the same way - on top of Three.js with the same polyfills (though slightly modified). What I didn't count on was that our approach to everything was so similar that it took me just a few hours to completely swap my entire foundational scene out for their
This blew my mind, because I had my own 3D objects and groups created with Three.js and the only tag I put on my HTML page was that
Actually, there were a few hiccups along the way, but given that I was shoving what I thought was a square peg into a round hole, two minor code changes are nothing.
\nMy approach is like so:
\nHave a "BaseApplication" ES6 class. This class would be extended for your application. It used to be that I'd create the underlying Three.js scene here in the class, but with A-Frame, I simply pass the
Like I said, two hiccups. First, my application wasn't rendering it's children and I didn't know how to pickup the render event every frame. Easy. First pretend it's an element by assigning an "el" property to the class and set it to playing:
\nthis.el = { isPlaying: true };\nNext, simply register this class with the A-Frame scene behaviors like this:
\nthis._ascene.addBehavior(this);\nOnce this behavior is added, if your class has a "tick" method, it will be fired:
\n/**\n* a-frame tick\n* @param time\n*/\ntick(time) {\n...\n}\n\nLikewise, any objects you add to the scene, whom you want to have these tick methods, simply add them to the behavior system in the same way.
\nIn the end my hefty BaseApplication.js class that instantiated a 3D scene, plugins, and polyfills, was chopped down to something 50 lines long (and I DO use block comments)
\nexport default class BaseApplication {\n constructor(ascene, cfg) {\n if (!cfg) {\n cfg = {};\n }\n this._ascene = ascene;\n this._ascene.appConfig = cfg;\n this._ascene.addBehavior(this);\n this.el = { isPlaying: true };\n this.onCreate(ascene);\n }\n\n get config() {\n return this._ascene.appConfig;\n }\n\n /**\n * a-frame tick\n * @param time\n */\n tick(time) {\n this.onRender(time)\n }\n\n /**\n * add objects to scene\n * @param grouplist\n */\n add(grouplist) {\n if (grouplist.length === undefined) {\n grouplist = [grouplist];\n }\n for (var c in grouplist) {\n grouplist[c].addedToScene(this._ascene);\n\n if (grouplist[c].group) {\n this._ascene.appendChild(grouplist[c].group);\n this._ascene.addBehavior(grouplist[c]);\n } else {\n this._ascene.appendChild(grouplist[c]);\n }\n }\n }\n // meant to be overridden with your app\n onCreate(ascene) {}\n onRender(time) {}\n}\n\nAs you might be able to tell, the only verbose part is the method to add children where I determine what kind of children they are: A-Frame elements, or my custom ES6 class based Object Groups.
\nSo, at this point I said to myself..."Great! I still really think markup is silly, but A-Frame has a team of people that will keep up with WebVR and will update the basics as browsers and the spec evolves, and I should just use their
Then, I hit ctrl-alt-i.
\nFor those that don't know, this loads the A-Frame visual inspector and editor. Though, of course, it won't save your changes into your code. Let me say first, the inspector got reallllllly nice and is imperative for debugging your scene. The A-Frame team is forging ahead with more amazing features like recording your interactions in VR so you can replay them at your desk and do development without constantly running around.
\nSo, when I loaded that inspector for the first time, I was disappointed that I didn't see any of my objects. I can't fault A-Frame for this, I completely bypassed their tags.
\nThat alone roped me in. We have this perfectly nice visual inspector, and I'm not going to deny my use of it because I can't be convinced to write some HTML.
\nAt this point, me and A-Frame are BFF's. But I still want to avoid a 2008 era jQuery mess. Well, turns out, 3D object instantiation is about as easy in A-Frame as it is with code. It's easier actually because tags are concise, where as instantiating materials, primitives, textures, etc, can get pretty verbose.
\nMy whole perspective has been flipped now.
\nThat's it. I'm actually amazed how responsive the 3D scene is for appending and removing elements. There's no "refresh" call, nothing. It just works.
\nI actually created a little utility method to parse JSON into an element that you could append to your scene:
\nsceneElement.appendChild(AFrameGroup.utils.createNode('a-entity', {\n 'scale': '3 3 3',\n 'obj_model': 'obj: ./assets/flag/model.obj; mtl: ./assets/flag/materials.mtl',\n 'position': '0 -13 0'\n}));\n\nAFrameGroup.utils.createNode(tagname, attributes) {\n var el = document.createElement(tagname);\n for (var c in attributes) {\n var key = c.replace(/_/g, '-'); // hyphens not cool in JSON, use underscore, and we convert to hyphen here\n el.setAttribute(key, attributes[c]);\n }\n return el;\n}\nYah, there's some stuff I don't like all that much, like if I want to change the position of an object, I have to go through element.setAttribute('position', '0 50 0'). Seems bit verbose, but I'll take it.
\nOverall, the markup aspect, early versions, and lack of organization/cleanliness in the code examples made me sad. But examples are just examples. I can't fault them for highlighting simple examples that don't scale well as an application when they intend to showcase quick experiences. A-Frame wants to be approachable, and if I yammer on to people about my ES6 class based approach with extendable BaseApplication and BaseGroup/Object classes, yes, I might appeal to some folks, but the real draw of A-Frame right now is for newcomers to fall in love with markup that easily gets them running and experience their own VR creations.
\nAll that said, I did want to share my experience for the more seasoned web dev crowd because if you peel back the layers in A-Frame you'll find an extremely well made library that proves to be very flexible for however you might choose to use it.
\nI'm not sure whether I want to link you folks to my library that's in progress yet, but I'll do it anyway. It's majorly in flux, and I keep tearing out stuff as I see better ways in A-Frame to do things. But it's helping me prototype and create clear distinction of helper code from my prototype logic, so maybe it'll help you (just don't rely on it!)
\n", "date_published": "2017-07-14T00:00:00Z" } , { "id": "https://benfarrell.com/blog/2017-04-17-back-from-vrla/", "url": "https://benfarrell.com/blog/2017-04-17-back-from-vrla/", "title": "Back From VRLA", "content_html": "I believe it was during a session called “Shooting VR for Post” that I found myself identifying heavily with one of the panelists who said something to the effect of “Before VR, my work was a bit mundane. We’d take a look at a shot we needed to do in a meeting, and we wouldn’t even have to talk, we’d instantly know what our roles were and break to get down to work. With VR now, it’s not that easy, we need to knock our heads against the wall and really come up with ways to get the job done.”
\nAs a web developer, I share this sentiment completely. The speaker expounded, giving an example like when Houdini comes out with a new node (I can only vaguely guess what this means), there’s a level of excitement, but it’s short lived. I feel similarly when a new Web API or Node.js based front-end workflow enhancement comes out, or a new framework is released. It changes our workflow in a nifty way, but it doesn’t necessarily change the work we create in a meaningful way.
\nIt’s a big sentiment, and I feel it’s absolutely monumental that I happen to share this sentiment about the same new technology with a cinematographer…someone whom I might never even speak to in a professional capacity. I also seem to share this sentiment with sound engineers, game developers, VFX artists, hardware manufacturers, and more. I even had a fascinating conversation about depth being registered in your hypothalamus vs your visual cortex with a Game Developer/Designer/Cognitive Psychologist.
\nI’m silo-ing people a bit here because the more curious amongst us (including myself) have always enjoyed exploring the fringes of our craft. It’s not necessarily true that I wouldn’t talk to a cinematographer as a web developer, but it’s also not necessarily normal.
\nThe point is that VR is bringing the best minds from all disciplines together and dissolving the fringes between these disciplines. Conferences like VRLA allow the stories of these boundaries breaking down to be told.
\n![\"IMG_20170415_131157\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2017/04/IMG_20170415_131157.jpg\")
This is incredibly important, not only for getting acquainted with what skills are being injected into this new medium and why, but also because nobody knows the right way to do things. When there’s no right way to do things, there’s no book you can buy, nothing to Google, nothing we can do except hear about fleeting experiences from people that got their hands dirty. We need to hear about their pain and about their opinions formed from creating something new and unique. When we hear lots of such perspectives, we can assemble a big picture, which I’m sure will be shattered by the next VRLA. I’ll be waiting to learn about the hypothetical magician a panelist cited as a great collaborator for focusing attention in a 360-degree world.
\nAlso interesting is the regionality of VR creators. I feel like I hear an entirely different story in San Francisco versus what I heard at VRLA. When I attend the (admittedly low number of, so far) meetups around the Bay Area, it’s mostly about hardware, platforms, new app ideas, prototypes, social experiences. In LA, I feel that it was overwhelmingly VFX, cinematography, sound design…a very heavy focus on well-produced content. I’m still uncertain about the regionality around game development, perhaps because it’s relatively regionless. Though, one memorable paraphrased line on that subject was “Game devs are now sitting in the same room as VFX artists and directors.”
\nPerhaps one of the more interesting things I picked up was the different stories from different creators on immersive video. Immersive or 360 video seems like a mainstay in VR. The cries of it not really being VR have been sufficiently drowned out with most, if not all, presenters acknowledging the sentiment but disagreeing with it. Andrew Schwarz of Radiant Images, for example, called immersive video the “killer app” of VR. I expected this sentiment, especially in a city with so much film talent.
\n![\"Andrew](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2017/04/IMG_20170414_152405.jpg\")
Andrew Schwarz of Radiant Images showing the new completely modular camera mount (AXA 360 Camera System) for immersive media
What I did not expect was the nuance verging on disagreement from Dario Raciti of OMD Zero Code. His point of view seemed to be that the novelty of immersive video has waned. His interest lies in creating marketing campaigns that make brands like Nissan and Gatorade stand out from the rest. Answering my question of what kinds of projects he tries to sell to clients, he flat out says he tries to discourage pure 360 video. Instead, he prefers a more immersive experience mixed with 360 video.
\nAn excellent example of this was his “Let Hawaii Happen” piece. The user begins on a parachute they can steer and navigate to various islands in Hawaii. Once they’ve landed, it switches to a non-interactive 360 video tour.
\nI think Dario’s take on advertising with VR is very much worth listening to. His team also created a car-shopping VR experience for Nissan in which the user is seated to get a feel for the interior of the car, much like what you would do car shopping in reality. Outside the windows, however, a much different scene plays out: the viewer is also part of a battle in the Star Wars universe.
\nThat exemplifies Dario’s notion of mixing real-time 3D content with immersive video, but it also touches on his point about advertising in general. To liberally paraphrase, Dario feels you should never beat the user over the head with branding. No logos, no mentioning of the brand unless its subtle and integrated into the experience. The experience always comes first, and if it’s memorable, it will sell the brand.
\nTo me, this speaks to the larger issue of taking concepts we already employ en masse in traditional media and shoe-horning them into VR. Advertisers, I know you’re already thinking of this. You want to cut to commercial, put your logo on the bottom third of the screen, and include voice overs about how your brand is the best. Dario is saying to create good marketing experiences, let the content flow freely and be subtle about your brand. Consumers will respond better. He even cited “Pearl,” an Oscar-nominated VR short, as an example of something that could be a commercial with extremely limited changes.
\nThe notion of shoe-horning brings another memorable notion to mind. To date, I’ve been thinking about VR like the jump from desktop to mobile. But the better analogy from one panelist was that “VR is like the jump from print to digital.” While stubbornness to hold on to the old ways can be detrimental, years of experience coupled with open-mindedness can be a huge asset.
\nIn the Cinematographers’ panel, it was mentioned that old 3D tricks, because of limited processing power, are now coming back into fashion. The reason being that game engines like Unreal are coming into favor for doing real-time previews of scenes. Even traditional film equipment is being recreated in VR to help production. To hear a cinematographer talk about replicating a camera crane in VR and then shrinking it down, scaling it up, putting it on a mountain-top…. all within a day’s shoot was incredibly interesting.
\n![\"IMG_20170414_173154\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2017/04/IMG_20170414_173154.jpg\")
Shooting VR for Post Panel
The panelists and presenters at VRLA shared so much of their recent, and super fascinating, experiences based on their experimentation. This was a bit unfortunate, because I found myself glued to the presentation rooms and out of the expo floor. I saved my 2-hour lap through the expo hall until the very end. As expected, the lines for the more interesting experiences were either too long or closed. I can’t fault VRLA or their exhibitors for this; it seems a standard downside of VR conferences. I would wager that the most popular experience was the Augmented Reality (Hololens) Easter Egg hunt. As I didn’t experience it, I’ll just leave you with a photo because it looks awesome.
\n![\"Microsoft](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2017/04/IMG_20170414_142354.jpg\")
Microsoft Hololens Augmented Reality Easter Egg Hunt[/caption]
Of course, like Microsoft, a bunch of big vendors were there: Facebook, HTC, Intel. Although I don’t own a Vive, their talk of the multi-platform subscription service and their wireless headset was exciting. So was hearing how dedicated Intel, HTC, and HP are to VR developers. Yes, Facebook and MS are dedicated to Mixed Reality as well, but for me, that message was well received a while ago, so it’s awesome to see the pile on.
\nBeing that there were around 170 exhibitors at VRLA, there were tons of smaller ones showing games, hardware, new experiences, and new creative tools. One notable company, Mindshow (http://mindshow.com), offers creative tools for recording animated characters with your body and voice in real-time. Watching from the expo floor, I was a bit disappointed as it felt too scripted. However, a booth attendant assured me it was that way for the 10-minute, quick demo for conference go-ers. It makes sense that you’d probably not want to start users with a blank slate if you only have a short window to impress them. So, if Mindshow is what I think it is, I can imagine having so much fun myself, and I can see many people creating awesome animated content extremely easily….but I’ve been known to overhype things in my own head.
\nThough it was my first time, VRLA has been going on for 3 years now and they’ve grown exponentially. The conference-going experience was not as seamless as others I’ve been to. The Friday keynote was delayed by at least 30 minutes because the speaker had no slide notes, which set off a cascade of presentation time pushbacks. There were constant audio issues, and the light field talk I was really looking forward to was cancelled with no explanation. This is all forgivable and probably par for the course given how many people from different disciplines are coming in and bringing their passions and experiences. There’s an amazing energy in VR. Organizations and conferences like VRLA focus it. It might not be laserlike as VR grows exponentially, but with a medium so young and with so many stories still to be told from creators about their experimentation, everything is appreciated.
\n", "date_published": "2017-04-17T00:00:00Z" } , { "id": "https://benfarrell.com/blog/2016-10-16-selfcomposed/", "url": "https://benfarrell.com/blog/2016-10-16-selfcomposed/", "title": "Self Composed", "content_html": "I was one of a few developers on this project. I definitely didn't lead it as MacOS development in Objective-C isn't\nmy forte, but I think I did lots with tying the print server to the UI and experience.
\nSelf Composed was a project my team and I at Adobe did for the San Francisco Museum of Modern Art and stayed up through\naround 2023.
\nUsers could create unique compositions with the installation's 2 cameras where one view masked the other. So walking up, you'd\nuse whatever was in your pockets, place it on the table and create a selfie you could share.
\nHere's the writeup from the SFMOMA, but also we went on to win\na Communication Arts Award.
\n![\"the](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/portfolio/SFMOMA-galleries-with-visitors_0551_April-29nd_-Henrik-Kam-2016-1024x683.png\")
\nwe made two installations, side by side (fabrication of the case, though was contracted out
![\"playing\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/portfolio/Playing_with_Self_Composed-576x1024.jpg\")
\nplaying with self composed
With the Oculus store continuing to get some interesting titles while I wait with baited breath over getting my very own Oculus Touch Controllers when they get released, its easy to forget about 360 Video. Some say 360 Video is not VR at all like Vive engineer Alan Yates:
\n![\"notvr\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2016/08/notvr.png\")
The problem Mr. Yates seems to have is that 360 Video chops off some key VR immersion factors like being able to move your body (not just your head) as well as being able to interact with the scene and have the scene interact with you.
\nIn addition to lacking some key things that make VR the immersive thing it is when you think about content like games, it can also box in the content creators to doing things that are less than ideal in VR. A big one here is knowing that when in VR, an experience can induce a little motion sickness by moving the user against their will. Another is dropping a user into a space leaving them disoriented and confused while they figure out where they are. 360 Video continues to suffer from this as traditional video creators make the choice to pan their cameras, or don't pay enough attention to their viewer when cutting between scenes.
\nAll that said, whether it is or isn't VR, it's certainly an emerging art form. It's enough like traditional video to be approachable for longtime video creators, but also deceptive in that these same creators need to question fundamental techniques or risk creating bad or even vomit-inducing experiences.
\nIt had been a while since I last spent a couple hours enjoying 360 Video, so yesterday afternoon I decided to go back in. Things have improved, and there are some fascinating shorts being produced as well as not so fascinating shorts with gimmicks that may or may not work. Let's talk about them.
\nRuntime: 13:03
\nFilesize: 586MB
\nKicking things off with Mr. Robot is a bit unfair. Not only do I love the TV show, but of the shorts I've seen so far I like this one the best. It uses a number of gimmicks that don't feel gimmicky, and breaks some other rules in a way that feels OK.
\nAlso interesting, is that many of the best videos that I've seen and I want to talk about are directed or co-directed by Chris Milk who runs a VR Storytelling company called Within (and then posted on the Within Oculus channel). Despite Mr. Milk making some compelling shorts, Mr. Robot shines greater than any of them for me AND it is directed by original Mr. Robot creator Sam Esmail.
\nBTW...only read the rest of this one if you can handle Season 1 spoilers.
\nElliot, the main character, turns out to have some major split personality disorder. Through season 1, Elliot's routinely expresses his thoughts to you, the viewer. It would be a bit gimmicky breaking the fourth wall like this all the time, except when you consider the possibility that the viewer is another personality or just a remnant of Elliot's troubled mind.
\n![\"elliotlookingatyou\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2016/08/elliotlookingatyou.png\")
Elliot acknowledging the user
The 360 short uses this setup to it's advantage, and as you might expect it just so happens to be perfectly suited for VR. You enter the experience next to Elliot on his couch while listening to his internal thoughts (expressed as a voice over) lamenting about some past memories. In true Mr. Robot fashion, Elliot looks at you to acknowledge your presence occasionally. Turns out this works great for VR, too. The user having a presence (even if just an inactionable one) does great for immersion.
\nAn interesting thing that happens early on is that the director wants to focus on the laptop in the kitchen. It's a bit weird, in that it feels like a throwaway story point that never really matters. That said, with traditional video, a director might edit the shot and cut over to the laptop. However, with 360 video we can't have hard edits like this that disorient the viewer, so instead the lights flicker in the kitchen around the laptop and the user's attention is drawn there.
\nElliot also happens to be smoking a joint which presents some interesting opportunity for gimmick. Elliot takes a big puff and exhales at the camera which offers an opportunity to transition to a past memory. While this isn't necessarily a 360 video gimmick, what follows is him sitting in the exact same spot in his past memory. In fact the scene looks no different, which is an important point not to disorient the user's experience. The use of whiting out the scene with smoke serves to transition the story but not necessarily the set.
\nThe marijuana use also provides a convenient way for the camera/viewer to get "high". As the marijuana starts taking effect, the camera starts floating to the ceiling offering a wider view of the shot and allowing Elliot's past love interest to enter. He even calls out "Oh wow, I got really high...look at us up here". Very important to reiterate here that Esmail coupled the camera transition with talking to the user about it in while SIMULTANEOUSLY pretending its part of the story.
\n![\"elliogettinghigh\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2016/08/elliogettinghigh.png\")
Camera floats up as Elliot gets high
To further layer on this storytelling feat, the camera view slightly shifts the user to look at the door in preparation for his former love interest Shayla to walk in.
\nAs Shayla knocks on the door, something a bit awkward happens which is that with each knock, the camera cuts slightly in every direction a few times. It feels like a mistake, but perhaps it was an opportunity to cover up a production mistake where the shots weren't aligned.
\nShayla enters and wanders a bit around the room talking to Elliot. As she enters the bedroom, a light turns on and illuminates the room for a moment. To me it was a bit awkward and I couldn't find any purpose to it, but its over quickly.
\nAs she wanders around the camera pans a bit, which is breaking a bit of a VR "rule" since you have no control over it - but it's done gently and after the initial camera floatation and other movements so doesn't feel wrong in the least. Here the full 360 comes into effect as Elliot stays on one side of you and Shayla walks behind you. You, the viewer are in the middle of this conversation and it feels like being in the middle of a group, turning your head each way to see each person speak.
\n![\"shaylaandelliot\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2016/08/shaylaandelliot.png\")
Shayla and Elliot talking
After the two leave there are some establishing shots of the beach, the shut down amusement park that would be later used as a base of operation. In these shots there is some gentle movement. Again, it seems to break a bit of the VR rule of not forcing the user to move in a way they aren't actually moving their body - but here it feels right and I think making it a short establishing shot that's not integral to on-screen dialog is the right way to go.
\nMore of the story happens when the video cuts to inside a ferris wheel car. As the video goes on, Esmail seems to be limiting the storytelling to slow paced enclosed areas with the dialog being a bit slow paced as well - more like what you'd find in real life, not fast moving heated dialog with fast cuts. Again, in the Ferris Wheel scene, you must turn to each character as they talk, much like you would do as the third and silent wheel in a real conversation sitting behind two people.
\n![\"ferriswheel\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2016/08/ferriswheel.png\")
Shayla and Elliot in the Ferris Wheel
What's interesting here, is that I did previously watch this 360 video on a monitor using my mouse to pan around. I thought it was a bit boring, and I didn't make it past the opening title, judging it as another property to jump on the 360 bandwagon. But, here in VR, what didn't work on a screen is a great storytelling experience.
\nIn the next scene, Elliot and Shayla are strolling along the boardwalk. Important to note here is that the camera is following them again, moving the user. Esmail didn't put any important dialog in this scene, only using the tone and mood to convey a story point (that Elliot and Shayla have made up after some cold-shouldering and are having a happy memorable time). I find this interesting to contrast with the slow pacing and slow conversations that are placed in static scenes. To get in Esmail's head a bit, I might be inclined he believes that the camera shouldn't be moving at all when you need the viewer to focus on a important bit of story. This scene itself transitions to a interesting colorful montage.
\n![\"Happy](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2016/08/montage.png\")
Happy montage
For sure, Esmail did lots of interesting things here. I'm sure I can rewatch again and again and find more. I do want to move on to other videos as interesting as Mr. Robot is. That said, I DO want to end Mr. Robot with one scene that really stood out for me, and that's when Shayla and Elliot are relaxing in bed. I am interested in 360 Video to offer perspectives that aren't seen in real life, and this bed scene touches upon that desire but doesn't go overboard. Check out the following shot with the camera positioned above the bed making them look upright. This is often used in TV/film and looks completely normal. In 360, however, it takes some getting used to. There's some dissonance while you figure out your orientation - but once you do, there's a bit of an aha moment that really adds to the experience. Other than the camera orientation, this scene is more of the slow, enclosed, conversational scenes that make the rest of the piece work well.
\n![\"inbed\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2016/08/inbed.png\")
Shayla and Elliot in bed
Runtime: 8:00
\nFilesize: 787MB
\nTo be clear, I really did like this video, which is surprising because I'm one of those people that think SNL has dropped way off in quality since
Since there's not much to say, its the perfect opportunity to note opening credits for most 360 Video I've seen. It's really no more than noteworthy, but opening credits and closing credits all seem to have standardized around displaying what they need to display in a narrow field of view, much like what you'd see on screen/TV and then replicating it 3 or 4 times over around your head so you can see the text and intro graphics regardless of where you look.
\n![\"SNLtitle\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2016/08/SNLtitle.png\")
Opening credits and corner of repeated title screen
With that out of the way, we can talk the actual video. In this experience, the 360 camera looks like it's positioned just above the traditional TV camera. You are facing the host, Jerry Seinfeld, and can turn to look at the crew and the audience...the entire room.
\n![\"Seinfeld](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2016/08/snlseinfeld.png\")
Seinfeld Hosting
If you've never been an audience member for SNL before (I haven't), it's interesting to experience what it's like behind the camera. You can see camera operators, a photographer, the boom mic crane, what the set looks like, etc. Its fairly interesting.
\nUnfortunately, the action takes place fairly quickly right away and you have to start paying attention. Contrast this with other VR stories and 360 video. Typically you might want to give the user some time to acclimate to the scene before starting the action. Here, being in the SNL audience is interesting, but Jerry Seinfeld is on stage in just 15 seconds and starts delivering his monologue so I was a bit torn what I want to pay attention to.
\nIf it was JUST watching a monologue intended for TV, this would be a disappointing experience. However, it turns into a sketch. Jerry launches into a Q&A with the audience who just happen to all be celebrities and/or past hosts.
\nYes, its funny. And YES it makes use of the 360 experience. The viewer is immersed into the audience here, because you watch Seinfeld call on somebody in the audience, and you turn to see if you can find them. In this way, the video sort of acknowledges that you're there by making you do what you would have to do in real life as an audience member.
\nHere's where things break down, though, and its purely technical. Check out this shot of when James Franco asks his question:
\n![\"James](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2016/08/franco.png\")
James Franco asks a question
Can you even tell that's James Franco? More importantly, do you think it would be easy to identify his facial expressions and body language? Recognition is so important to this bit involving celebrity. And facial expressions and body language are key to comedy and acting. You might think this is an anomaly because he's a bit far away. After all, John Goodman is another feature and he's fairly recognizable but also fairly close (hint: he's just past the lady at the bottom center). Its a fair point if you're just looking at this image, but in the experience Franco looks fairly close...just blurry and not crisp enough in the encoding. As a viewer you feel like you SHOULD be able to see him better and it's imperative to the experience, but the detail of the capture and/or the encoding prevents this.
\nOddly enough, Mr. Robot didn't suffer from this despite being longer and having less overall file size. This point is exactly why I'm prefacing each writeup with the duration and file size. This SNL video is closer to what you might expect from live 360 shooting without the benefit of planning and script to overcome these type of issues.
\nLast disappointing bit is that while it's interesting to see camera operators, the boom mic, etc, you can ALSO see the folks that hold the cue cards. It really detracts from the spontaneous and hilarious premise of this off the cuff Q&A to have cue cards right in front of you.
\n![\"cuecards\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2016/08/cuecards.png\")
Center: Assistant holding a cue card for Larry David's question
All in all, this is a fairly successful 360 video. I quite enjoyed it, but where it falls down, it suffers because 360 isn't really the medium they intended to target with this bit.
\n\nRuntime: 8:08
\nFilesize: 1.07GB
\nVice News, from my understanding, is a bit more gritty and risky/risque than traditional news media. When I come across Vice stories and videos I seem to recall reporters inserting themselves into dangerous scenes and doing raw reporting. Even I've put Vice in the wrong nutshell, that seems exactly what they are doing in this 360 video (though to be fair, a protest esp in NYC is probably not going to be dangerous at all). This protest in particular is to highlight the unfair treatment of black men and women at the hands of the US justice system and police.
\nOne interesting thing done right off the bat is in the opening title. I criticized SNL's 360 video for not giving enough time for the viewer to get acclimated to the scene. Here it's just right, and they've incorporated the title of the piece in an interesting way. It's wrapped around your head with most of the title outside your field of view at all times. So, to read it, you must look all the way to the left and pan your head from left to right. Meanwhile, a crowd (protesters) appear below you.
\n![\"Vice](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2016/08/vicenewstitle.png\")
Vice News Title Screen
Initially I panned this as a bad choice. But, after thinking about it, having to work across 270 degrees to read the text doubles as a mechanism to also take in the scene around you. Given that an Oculus user already had to click on the video thumbnail and download, having the title be legible again in the video is not as necessary as one might think. So, even if the user fails to read and struggles to take all the text in in one go, its still OK.
\nAfter the opening scene, we cut to being eye level right next to a demonstrator. This group of men chants "I can't breathe with y'all on my neck", of course a reference to the killing of Eric Garner.
\n![\"Demonstrators](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2016/08/cantbreath.png\")
Demonstrators chanting "Can't breathe"
What was interesting for me is my reaction to being right up next to the demonstrator. In real life, I tend to stay far away from demonstrations like this, whether it be Black Lives Matter, a subway musician, or someone in a city park on a microphone calling for sinners to repent. Reflecting, I think it comes down to one thing for me: how to react and what kind of body language am I using in response. For example, someone standing on a soap box saying the world will end tomorrow is super interesting. I don't take them seriously, of course, but I would love to hear their whole speech for the entertainment value. On the opposite end of the spectrum - a demonstrator like this, representing a real cause, I might like to hear what they are saying, but especially with myself being white and someone who historically COULD look down on them, I may be a bit self conscious of what type of message I'm sending to them with my reactions (or lack thereof) as I stand there and observe them.
\nI talked before about how 360 videos do well to acknowledge the viewer as part of the scene. Mr. Robot does this exceptionally well, and SNL with Seinfeld did this to a small extent. In a scene like this, NOT acknowledging the viewer seems to work exceptionally well. I can observe and actually listen to the message without worrying about my reactions or being self conscious.
\nIn addition to watching demonstrators, I've never been part of a march. So it was interesting to be swept down the street passing chanting, banners, stalled traffic etc. As this is a camera, regardless of whether it's 360, if it's being carried somewhere, there needs to be an operator. While the cameraman usually stays perfectly behind the viewer through most of the video, he's bit close for comfort in this scene:
\n![\"cameraman\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2016/08/cameraman.png\")
Cameraman, a bit too close for comfort
Like I said, its a bit disconcerting, but its 360 footage being capture in an uncontrolled environment. He can hardly be blamed!
\nIn the next scene, we follow a reporter down the street to a "Die In". A few dozen people are lying on the ground to demonstrate dying in the streets. Unfortunately, the technology and more specifically the capture resolution/encoding failed here much like it did in the Saturday Night Live video. For starters it was night time, so the visibility wasn't great, and well...can you tell what is actually happening in this scene?
\n![\"diein\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2016/08/diein.png\")
A "Die In" demonstration
This image, as in VR is extremely hard to comprehend. Its actually worse in VR because you feel like you're there and because of that you think you SHOULD be able to pick out the shapes on the ground as people and bodies. I was actually a little surprised when they started getting up because some of those people were white. I convinced myself that part of the reason that I couldn't make heads or tails of the scene was that a black man or woman's face at night lying on the ground would be surely be hard to see. But no in fact, there were many white men and women as well.
\nI'll end this video's analyzation with an interesting reaction I had to one of the protestors. In the same "Die In", as people were starting to get up, one lady got up half way, raising her hands and ended up on her knees. The reporter we were following crouched down next to her to interview and get her thoughts.
\n![\"crouched\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2016/08/crouched.png\")
Reporter crouching to interview a demonstrator
What was interesting for me was my posture as this happened. Previously, I was sitting upright in my office chair as I watched this video. However, when the reporter crouched down and the camera followed, my entire posture in my chair went lower into a seated squat. I took note that it was an interesting thing that with enough direction from on screen cues, my body would follow!
\n\nRuntime: 8:09
\nFilesize: 481MB
\nCatatonic was a fun little horror experience akin to what you'd get when you and your friends drive for a couple hours on Halloween to a small rural town where someone has redone their barn or farm to be a spooky experience complete with actors jumping out at you to scare you.
\nThis 360 video takes place in a run down insane asylum. Despite thinking it worked pretty well, it did what contemporary VR creators dictate you should not do: put the camera on wheels and roll around. I eluded to this before when some of the videos above had this to a lesser effect, and it harkens back to early VR tests when lots of people experimented with putting you on some kind of track like a rollercoaster. The movement depicted in your vision contrasted with the lack of movement felt in your body was named as the prime reason for feeling motion sick. So, of course, content creators name this as a prime thing not to do.
\nAll that said, I personally felt fine despite being on a moving track the entire time. In the story, you are a patient being wheeled through the asylum in a wheel chair. In addition to being slow, you can also look down and see your body and chair. Perhaps this addition of a "reference object", or something that persists in the same place in your field of view, cancels out or minimizes the motion sickness.
\n![\"In](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2016/08/catatonic-lookingdown.png\")
In a wheelchair (reference object to reduce motion sickness?)[/caption]
Remember I talked about those Spooky Barns? Well, some people get scared by the things jumping out at you. Not me (or my wife for that matter), we see things coming maybe get a little surprised, but really just end up giggling at the attempt. Same here. The first thing you encounter is this zombie looking girl that kinda snarls and lunges at you as you roll past. I had the same reaction. Ironically, I was much more concerned as I was wheeled into the room that my knees would smash into the doorway (no seriously, it made me a bit uncomfortable).
\n![\"Scary](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2016/08/scarygirl.png\")
Scary girl
All in all, it was more of the same. Interesting and fun no doubt, but not TOO much more noteworthy to speak of. I was wheeled past disturbing patients. Tricks of light and time dilation made things creepier as well.
\nOne thing that really made me take notice of after I experienced it was the use of time and making me impatient to look around. There is a quite normal looking man that wheels you around for the first half of the experience. He even tries to soothe you in the beginning. But, he's behind you. It's an effort to look backward and up to take note that there's someone there. I think I only did it once out of curiosity.
\nHowever, an interesting thing happened. After a particularly fast paced period of time when lighting suddenly changed and time sped up for a second and things got creepy, there was a few seconds of complete inaction. I was left sitting in the chair standing still and nothing was happening. The scene had the effect on me to make me impatient and look behind me to figure out why I wasn't moving. It turned out the nice man was gone, and a scary hooded figure lurched out a door and took over. If I wasn't given time to get impatient (possibly after such an actionable few seconds), I would not have looked backwards (again it's awkward to do so) to see what I was supposed to see.
\nFrom there, the cheesy horror and effects picked up!
\nIn real life I've been getting bloodwork lately, and I JUST CAN'T look at the needle going into my arm. It's too much...freaks me out. However, when presented with the following:
\n![\"needles\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2016/08/needles.png\")
...I can't look away! I know it's not real, so I guess I feel empowered to watch it and laugh off my complete irrational freak out on needles.
\nAnd from then on its more good, cheesy horror with some personal bubble invasion thrown in to try to make you uncomfortable.
\n![\"Invading](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2016/08/personalspace.png\")
Invading personal space
So that's Catatonic! I figure if those cheap horror movies and Halloween barns scare you, this might as well. For me, it was good B-Movie fun.
\n\nRuntime: 2:17
\nFilesize: 159MB
\nThis is a quick one! It has a bit of a novel approach, though. I'm not sure how well it works to be honest with you. The video opens on a beach. In front of you is a washed up boat. Really just a nice relaxing scene, and it holds there for around 40 seconds for the viewer to get acclimated. Again this seems fairly standard practice for VR Storytelling.
\nPrior to the narrative starting, a lady walks her dog directly in front of your view. My first couple of times through, the dog walking seemed a bit meaningless and odd. I ignored it waiting for something ELSE to start up. It turns out though, on my 3rd viewing, I noticed its a guiding action. It was a bit of action meant to make you follow your head behind you where the actual narrative started with the two main characters.
\n![\"\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2016/08/openingscene.png\")
Walking the dog. A guiding action to the main story
So obviously this bit of direction missed the mark for me. Luckily, I hear a voice over narrative, and I know to look around for what was going on.
\nThe interesting bit about this experience is the spatial audio. The setup is that this couple is fighting and go off to different areas of the beach. You can see each by turning your head, but also when you turn your head you can hear each of their thoughts...a narrative of their anger towards the other from their perspective.
\n![\"Split](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2016/08/splitscreen.png\")
Split View
Unfortunately, I didn't think this worked so well, because it took a long time in the short span of this video to figure out that there was different audio depending on where I looked. When I figured it out, I got a bit frustrated because I couldn't listen to both dialog at once and felt like I was missing things.
\nAll that said, it was an interesting device to tell the story!
\n\nRuntime: 5:20
\nFilesize: 273MB
\nLoVR is an interesting concept. Its all computer generated data visualization that you fly through, and it's about love. Aaron's description verbatim is this:
\n\n\nA story of love, told through neural activity. Chemicals are released, neurons are activated and a form of poetry is revealed within the data.
\n
You know, to be perfectly honest, I'm not quite sure this needs to be in VR. I dig the concept of measuring the brain's neural activity and pinpointing the moment that falling in love happens. At that moment the music picks up and the dataviz starts getting extreme.
\nI want to guess that this experience was done with VR in mind, but the creator wanted to expand reach to flat screens as well so made an experience that could encompass both. Flying through the visuals is a nifty experience, but at the same time, not much of your periphery or even behind you matters.
\nAll that said, its a nifty concept and video!
\n![\"Baseline](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2016/08/notinloveyet.png\")
Baseline reading, not in love yet
![\"Flying](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2016/08/flyingthrough.png\")
Flying through, noticing beauty and starting to sweat a bit - music is picking up
![\"Looking](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2016/08/lookingback.png\")
Looking back on the chaos at the moment of love
Runtime:4:56
\nFilesize: ???
\nI'll end this post with a weird one. Despite having some negative comments on various aspects on all the 360 videos I talked about, the criticism is just to point out interesting decisions and aspects. Overall, the videos I watched were pretty damn great. 2016 is just the tip of the iceberg as well. 360 video will continue to evolve as an art form and I think we're mostly in the experimental stage of things right now. All of the above videos were from Within, and its certainly no mistake that a company founded on great VR Storytelling would produce and highlight great 360 video.
\nWhat I'm about to mention next isn't anything like that, but it has a unique take on instructional videos!
\nI've been to Lowes Home Improvement stores before for various projects, and they really do try to help guide you through your projects. Their website is no different. Having owned a home, I've gone through some of their instructional videos or tutorials a few times to make or fix something. It does help, for sure.
\nHowever, when your hands are full and you're trying to fix something while at the same time trying to scrub a video back or paginate the instructions back because you missed something...well it's a pain!
\nThis video attempts to address that problem. I question the effectiveness as I wonder how unweildly wearing a headset (even if wireless like the GearVR) would be while trying to do home repair. All the same, its a bright idea!
\nThis instructional video is to make a quick DIY cement table with wooden legs. Instead of introducing the steps over time, the steps are done in space. So step #1 is right in front of you. As you turn your head in 360 degrees you can see the rest of the steps. This makes it easy to go back and forth between steps you might not understand...just look the other way! Each step video is on a continuous loop so the action is incredibly easy to revisit.
\n![\"Making](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2016/08/lowes.png\")
Making a table
![\"Finished!\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2016/08/finsihed.png\")
Finished!
This was a bigger blog post than usual, but it forced me to think through some very interesting pieces and evaluate what's good, what's bad, and just where we are technically and experience wise in 2016 for 360 video. I picked the few that I though were most interesting - so everything here I enjoyed and send my kudos to the creators. There are, of course, even more that are as interesting - but lots that fall flat as well. The most important thing to note is the fact that everyone is experimenting with what works and we are at the beginning of a new way of thinking about video. It will be quite interesting to see the caliber of stuff that 2017 brings!
\n", "date_published": "2016-08-20T00:00:00Z" } , { "id": "https://benfarrell.com/blog/2016-06-17-a-week-at-the-hololens-academy/", "url": "https://benfarrell.com/blog/2016-06-17-a-week-at-the-hololens-academy/", "title": "A Week at the Hololens Academy", "content_html": "Ahhhhh the Hololens. I finally get to check it off my list. When I express my disappointment with not being able to try it out to my friends and co-workers that are interested in VR, it's kinda like talking about going to Hawaii. "Ohhhh, you haven't been? You really should, it's an enjoyable experience." (said, of course, with a knowing smirk and possibly a wink).
\nThere's a good reason for that knowing wink. Its a massively cool device, and despite being publicly available now to early adopters, there's a waiting list and it's $3k. Someone mentioned to me that they are in the "5th Wave" of wait list. So, right now, it's hard to get your hands on it. And that's IF you're willing to shell out the money.
\nShould you buy it if you get the chance? Maybe. For me, there's lots of parallels to Google Glass from a few years ago, but also lots of reasons it might break free from technological oddity into the mainstream.
\nIn terms of sheer impressiveness in hardware, hell yes it's worth $3k. Though it can be tethered via USB for the purposes of big deployments of your project, it's completely wireless and independent. The computer to run it is built right into the device. It packs Wifi, 64GB of memory, a camera (both RGB and depth), and other sensors for headtracking (probably an accelerometer and gyroscope). Even the casing of the device is impressive. It looks slick, true, but the rotatable expandable band that makes every effort to custom fit your head is practically perfect. I originally didn't put it on my head completely correct at first, and the display was resting on my nose a bit which would have been uncomfortable after awhile. Turns out, if you balance it on your head correctly, it barely touches your nose and almost floats on your face.
\nCompare the hardware to something like the Oculus Rift or the HTC Vive which are just display and you supply your own computer to tether to (and aren't augmented reality). They run $600-800 plus at least a $1k desktop computer. I can't recall who, but someone with me made the almost cruel observation of the size of an NVIDIA GTX970 graphics card compared to the size of the entire Hololens headset.
\n[![\"nvidiavshololens\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2016/06/nvidiavshololens.jpg\")
The display is another massively cool hardware piece and makes the entire system come together as one. It has it's problems which I'll get into (cough cough field of view), but I'll talk about that in a second when I get to usability. And make no mistake....usability is why or why you should not run right out and purchase one of these devices. The Hololens isn't so much a tool as it is an experience. It's not a hammer and nail. It's more of a workbench. A beautiful workbench can be amazing, but if you can't open the drawer to get to your hammer and nails and you want to create something, it's worthless.
Awful analogies aside, and usablility aside, let me say a quick word about the training. Microsoft calls it "The Hololens Academy". It occurs to me just now, that this might be a thinly veiled StarTrek reference. In fact, ALL of the training assets were space themed. From a floating astronaut, to a virtual futuristic tabletop projector, to a mid-air representation of our Solar System.
\nMy company, Adobe, was kind enough to send me last minute to Redmond and do some learning. I honestly didn't know what to expect because it was so last minute. Was it super secret stuff? No...but considering I hadn't seen the not secret stuff yet, it really didn't make too much difference. In fact it was SO not secret that our class followed along with well developed training material that MS has published online.
\nIn fact, in a testament to how well developed it is...I was weirded out a bit on the first day to be honest. It had that theme park feel. Or that historical city tour feel. You know, where every word and joke your guide says is rehearsed and feels forced? But I got over that real fast, you know why? Because the sessions went like clockwork. The instructors kept exact time to an eerie degree, and the assistants WERE psychic. Virtually every time I had trouble, an instructor was behind me within a few seconds helping me out. I didn't raise my hand, look confused, nothing. And there wasn't a single time where I felt like they were annoyingly hovering. They just showed up out of the blue being insanely helpful.
\nThe room itself was laid out extremely well for training. An open workspace with large screen TV's on the wall facing every which way with the instructor in the center on a headset made a very great training space. The instructor didn't even drive the software. He or she (they changed out every 3 hours), would have someone else driving the presentation machine while they spoke. This kind of coordination takes practice, no doubt.
\nThe walls and tables were decorated for the event too, along with coffee tables specifically for placing your virtual assets on (holograms). The room is probably a permanent fixture specifically for this.
\nThis all means one thing to me. We've got publicly available training materials, with tons of care put into creating them, extremely well staffed and smart trainers, and a training room just for the Hololens. Add to this the hundreds of engineers working on Hololens, adding the fact that MS is just now offering developer support for it... and the message is loud and clear. Microsoft is placing a HUGE bet on the Hololens. They aren't half assing this like a lot of companies in their position might for a product that is so different and hard to predict how well it's adopted.
\nTraining style aside - I found another thing extremely interesting about the training. It's all about Unity.
\n\nUnity seems like kind of an underdog at the moment. It's essentially a 3D authoring environment/player. It doesn't nearly have the reach of something like Flash or Quicktime which at one point or another has been ubiquitous. Yet, its a favorite of 3D creators (designers and devs) who have the desire to easily make 3D interactive experiences. The reach of Unity alone (browser plugin, WebGL, Android, iOS, desktop application, Oculus, Vive, Gear, and now Hololens as well as others) puts it right in the middle of being THE tool for creating VR/AR/Mixed Reality content.
\nI was naive to not expect MS using Unity for experience creation. But, the fact is, it's one of the ONLY tools for easy interactive 3D scene creation. I honestly expected Microsoft to push us into code only experience creation. Instead, they steered us into a combo of 3D scene building with Unity and code editing (C#) with Visual Studio. To be honest I'm a little resistant of Unity. Its not that its not an excellent tool, but I've gone through too many authoring tools that have fallen out of favor. This training is a wakeup call, though. If Oculus, Gear, HTC Vive weren't enough to knock me over the head - a major company like MS (who has a great history of building dev tools) using a third party tool like this....well consider me knocked over the head and kicked in the shins.
\nThe exercises themselves, were a mix of wiring things up in Unity and copying/pasting/pretending to code in Visual Studio. Its a hard thing to build a course around especially when offering this to everyone with no prerequisites, but MS certainly did a good job. I struggled a bit with C# syntax, not having used it in years, but easily fell back to the published online material when I couldn't get something.
\n\nOK so, the Hololens has the sweet sweet hardware. It has the training and developer support. All good right? Well no, there's another huge consideration. The hugest consideration of all. How useable is it, and what can end users do with it?
\nYou might guess that what end users do with it is up to you as a developer, and that's partially right. Everything has limitations that enable or inhibit potential. Here's the thing, though - take the iPhone or iPad for example. When it came out it WAS groundbreaking. But it wasn't SO different that you had to experience it to imagine what it could do. Steve Jobs could simple show you a picture of it. Yep it had a screen. Jobs could show you interaction through a video: Yep you can swipe and tap and stuff. People were imaginitive enough to put 2 and 2 together and imagine the types of things you could do based on never having used the device. Sure, people are doing amazing things with touch devices that would have never been imagined without using it - but the simplest of interactions you can certainly get the gist when seeing it used without using it yourself.
\nVR is somewhat harder to pin down, but again, its somewhat easy to imagine. The promise is that you are thrown into another world. With VR, your imagination can certainly get ahead of itself. You might believe, without donning a headset that you can be teleported to another world and feel like you're there.
\nWell, yes and no, and it's all due to current limitations. VR can have a bit of a screen door effect meaning if you focus hard enough you feel like you're in front of a screen. With VR, you are currently body-less. When you look down you'll probably see no body, no hands, or even if it's a great experience, it won't look like YOUR body. This is a bit of a disconcerting experience. Also, you DEFINITELY feel like you're wearing a headset. So yes...with VR, you ARE transported to a different and immersive space, however you need to suspend disbelief a bit (as amazing as it is).
\nAR is similar but a little worse. I can only comment on the Hololens, but its not the magical mixed reality fairly tale you might be led to believe. Even worse MS's published videos and photos show the user being completely immersed in holograms. I can't really fault them for this, because how do you sell and show a device like this that really must be worn to experience?
\n\nThe biggest roadblock to achieving this vision is field of view. From what I've heard, its the one biggest complaint of the Hololens. I heard this going in and it was in the back of my head before I put the device on, but it took me an embarassingly long time to realize what was happening. A limited field of view means that the virtual objects or Holograms only take up a limited portion of the "screen". Obviously. But in practice, this looks totally weird especially without some design trick to sweep it under the rug and integrate the limitation into the experience.
\nWhen you start viewing a 3D scene, if things are far away, they look fantastic! Well integrated with your environment and even interacting with it. Get closer though, and things start falling out of your field of view. Its as if you're holding a mobile screen up fairly close to your face, but the screen has no edges and it doesn't require your hand to hold it up. Well, what happens to things off screen? They simple disappear, or worse they are partially on screen but clipped to the window.
\nI took this image from a winbeta.com article about the field of view, here's their take on it, but for our sake right now, here's a great approximation of what you would see:
\n![\"hololens-fov-2-29\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2016/06/hololens-fov-2-29.jpg\")
People also use peripheral vision to find things in a large space, but unfortunately in this scenario you have no periphery - so it can be easy to not have a good understanding of the space you're in right away.
\nThere are a couple other visual limitations that make your holograms a bit less believable. For one, you can certainly see your headset. The best way to describe it is that you can certainly see when you're wearing sunglasses and a baseball cap (though the Hololens certainly doesn't protrude as far as a cap rim). You can also see the tinted projection area and some of the contours of that area in your periphery. It easy to ignore to an extent, but definitely still there. Also, you can see through the Holograms for sure. They're pretty darn opaque, but they come across as a layer with maybe 90% transparency.
\nAnother point is that in all the demo materials, if you get suspiciously close, the object starts disappearing or occluding. This is directly due to a camera setting in Unity. You can certainly decrease this value, however even the lowest setting is still a bit far and does occlude, and even then, the Hololens makes you go a bit crosseyed at something so close. You might say this is unfair because its simply a casualty of 3D scenes. To that, I say to check out the Oculus Rift Dreamdeck and use the cartoony city demo. You can put your head right up next to a virtual object, EXTREMELY close, and just feel like you can touch it with your cheek.
\nLastly, overhead lights can cause some light separation and occasionaly push some rainbow streaks through your view especially on bright white objects like the Unity splash screen. This point, I can directly compare this to the flare of white objects on the Oculus Rift due to longer eyelashes.
\nFor these above reasons - I don't think the Hololens can be considered an immersive device yet like VR is. VR is really good at transporting you to a different place. I thought the Hololens would be similar in that it would convincingly augment your real world. But it doesn't for me. It's not believable. And thats why for now (at least 10-15 years), I'm convinced that AR is NOT the next generation after VR. They will happily live together.
\nIf VR is the immersion vehicle - something that transports you, what's AR? Or more specifically, the Hololens? Well, just because something isn't immersive, doesn't mean it can't be incredibly useful. And I think that's where the Hololens lies for the near term. It's a productivity tool. I'm not sure I think games or storytelling or anything like that will catch on with the hardware as it is now (as cool as they are demo-wise until the immersion factor improves). No - I think it can extend your physical screen and digital world to an exceptional degree. Creating art, making music, even just reviewing documents can all be augmented. Your creation or productivity process doesn't have to be immersive, just the content you create.
\nI think this point is where AR really shines over VR. In VR, we're clumsily bringing our physical world into the virtual world so we can assist in creation using things modeled after both our real tools and 2D GUI tools. And usually this doesn't work out. We have to remove our headset constantly to properly do a job. With AR, the physical world is already there. Do you have a task that needs to be done on your computer or tablet? Don't even worry about removing your Hololens. Interact with both simultaneously...whatever. In fact, I think one HUGE area for the Hololens to venture into is the creation of immersive VR content itself. One for the immersive, one for the productive.
\nThat's not to say I don't think casual consumers or others will eventually adopt it. It certainly could be useful for training, aid in hands free industrial work, anything that augments your world but doesn't require suspension of disbelief.
\n\nHololens immersion isn't all doom and gloom though. Spatial awareness is, in fact, AMAZING. The 3D sensor is constantly scanning your environment and mapping everything as a (not fantastically accurate but damn good) mesh. Since it uses infrared light like the Kinect to sense depth, it does have its limitations. It can't see too far away, nor super close. The sun's infrared light can also flood the sensor leaving it blind. One fun fact that I've learned is that leather seems to not reflect the light too well, so leather couches are completely invisible!
\nWe did a really simple demo of spatial mapping. It looked amazing how we lined the real walls with a custom texture with blue lines. My Adobe colleague decided to make the lines flash and animate which was super mesmerizing. Unfortunately, I didn't find the mixed reality video capture feature until after that, so here's a nice demo I found on YouTube of something similar (though a bit more exceptional and interactive)
\nhttps://www.youtube.com/watch?v=JPMUuaeyy0g
\nAs scattered IR light tends to be sort of...well...scattered, meshes certainly don't scan perfectly. That's fine because MS has built some pre-packaged DLLs for smoothing the meshes out to flat planes and even offers advice on wall, ceiling, floor, and table finding.
\nOf course, once you've found the floor or surfaces to ineract with, you can place objects, introduce physics to make your Hologram interact with real surfaces (thanks Unity for simply collision and rigid bodies!), and even have your Holograms hidden behind real things. The trainers seemed most eager to show us punching holes in real objects like walls and tables to show incredible and expansive virtual worlds underneath. Again...though...the incredible and expansive can't be immersive with the field of view the way it is.
\nHere's a good time to show our group lobbing pellets at each other and hitting our real world bodies. The hole at the end SHOULD have been on the table, but I somehow screwed up the transformation of the 3D object in Unity, so it didn't appear in the right spot. It does show some great spatial mapping, avatars that followed us around, and punching a hole through reality!
\nhttps://youtu.be/meH96gNPxCw
\n\nSpatial audio is another thing I'm on the fence about. It's a bit weird on the Hololens. I give Microsoft major props for making the audio hardware AUGMENTED but not immersive. In VR systems, especially the Oculus Rift, you'd likely have over the ear headphones. Simple spatial audio (and not crazy advanced rocket science spatial audio) is limited to your vertical plane. Meaning, it matches your home stereo. Maybe a few front sources (left, right, and center), and a couple back source on your left and right. With these sources, you fade the audio between the sources and get some pretty awesome positional sounds.
\nOn the Hololens, however, the hardware speakers are positioned above your ears on the headband. They aren't covering your ear like headphones.
\n\n[![\"0e9693c5-78a1-4c51-9331-d66542e5fee9\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2016/06/0e9693c5-78a1-4c51-9331-d66542e5fee9.jpg\")
So yes, you can hear the real world as easily as you could without the headband on, but being positioned above your ears make it sound like the audio is always coming from above. One of our exercises included a Hologram astronaut. You'd click on the astronaut, and he'd disappear, but he'd talk to you and you were supposed to find him. Myself and everyone near me kept looking up to find him, but he was never up high - and I'm sure this is a direct result of the Hololens speaker placement. I asked the instructor about positional audio that included vertical orientation as well, and he said it was hard computationally. I know there are some cool solutions for VR (very mathy), but I'm skeptical on the Hololens. The instructors did say to make sure that objects you'd expect higher up (like birds) appear higher up in your world. I personally think this was a design cop-out to overcome the hardware.
Last thing I want to cover is input. Frankly I'm disappointed with EVERYONE here (except for the HTC Vive). It seems mighty trendy for AR and VR headsets to make everyone do gaze input, but I hate it and it needs to die. The Hololens is no exception here, it's included in all the training material and all of the OS interactions. Same goes for casual interactions on the Oculus Rift (gaming interactions use an XBOX controller, still dumb IMO) and Google Cardboard. The HTC Vive and soon the Oculus Rift will have touch controllers. Google Cardboard will soon be supplanted by Daydream which features a more expressive controller (though not positional). I've heard the Hololens might have some kind of pointer like Daydream, but I've only heard that offhand.
\nGaze input is simply using the direction of your eyes to control a cursor on screen. Actually, it's not even your eyes since your eyes can look around....Gaze input is using the center of your forehead as a cursor. The experience feels super rigid to me, I'd really prefer it be more natural and allow you to point at something you aren't looking at. With the Oculus Rift, despite having gaze input, you also have a remote control. So to interact with something, gaze at it and click the remote.
\nThe Hololens on the other hand, well it SEEMS cool, but it's a bit clunky. You're supposed to make an L with your thumb and index finger and drop the index finger in front of you (don't bend your finger, or it may not recognize the action). You also have to do this in front of the 3D sensor, which doesn't sound bad, but it would be way more comfortable to do it casually on your side or have your hand pointed down. And to be fair, spoken keywords like "select" can be used instead. We did also play with exercises that tracked your hands position to move and rotate a Hologram. All the same, I really think AR/VR requires something more expressive, more tactile, and less clunky for input.
\n\nAll that said, the Hololens is an amazing device with enormous potential. Given that Microsoft's CEO claims it is a "5 year journey", what we have right now is really a developer preview of the device. For hardware, software, and support that feels so polished despite interaction roadblocks, it will be most likely be amazing what consumers get in their hands 5 years from now. So should you shove wads of cash at MS to get a device? Well, me...I'm excited about what's to come, but I do see more potential for VR growth right now. I'm interested in not just new interaction patterns with AR/VR, but also about exploring how immersiveness makes you feel and react to your surroundings. The Hololens just doesn't feel immersive yet. Additionally, it seems like the AR/VR community are really converging on the same tools, so lessons learned in VR can be easily translated to AR (adjusting for the real world aspect). The trainers made sure to point this out - the experiences you build with Unity should be easily built for other platforms. It will also be interesting to see in the next 5 years where Google takes Tango (AR without the head mounted display) and potentially pairs it with their Daydream project.
\nAll that said, it's all about use cases and ideas and making prototypes. If a killer idea comes along that makes sound business sense and specifically requires AR, the Hololens is pretty much the only game in town right now, so if that happens I'll be sure to run out and (try to) get one. But in terms of adopting the Hololens because of perceived inevitability and coolness factor? I might wait.
\nBut if you don't own any AR/VR devices, cant wait to put something in the Windows store, can live with the limitations, and are already an MS junkie - maybe the Hololens is for you!
\nI'd like to give a big thanks to Microsoft for having us in their HQ and having such fantastic trainers and training material. I expect big things from this platform, and their level of commitment to developers for such a new paradigm is practically unheard of.
\n", "date_published": "2016-06-17T00:00:00Z" } , { "id": "https://benfarrell.com/blog/2016-05-10-the-oculus-rift-dream-deck/", "url": "https://benfarrell.com/blog/2016-05-10-the-oculus-rift-dream-deck/", "title": "The Oculus Rift Dream Deck", "content_html": "Like I said in my last post, the "Dream Deck" is one of several free pieces of content to help you discover what VR is like. The deck itself, is a handful of mini experiences that last a few minutes and then fade to black inviting you to experience the next thing.
\nI also described the input mechanism here as a bit archaic. In fact, through much of the more simpler and/or utilitarian UI (like the video player), the "head cursor" is how it works for now. How it works, is that you'd look at the button/item you want to select - aligning a small subtle dot over the item, and then use the Oculus Remote to click on the item.
\n![\"dreamdeck\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2016/05/dreamdeck.png\")
I want to talk about a few of my favorite experiences in the deck. Before that, I did want to say that they are all varying degrees of nifty when you experience them. And also before my favorites, I want to talk about a couple of notable experiences. One is simply you standing in front of a very alien looking alien. Another puts you in the middle of a couple of robotic arms doing all sorts of loud and frantic mischief. These two experiences, while not the best of the breed for me, DO explore a concept unique to VR - and that's making you uncomfortable without necessarily doing anything expressly so.
\nWith both the alien and the robotic arm, you experience a closeness to something you as the user can't anticipate how it will act. You can imagine seeing both in the real world, but having it before you in VR is something new. Both are extremely close to your virtual presence. The robot's frantic activity and the alien's lack of activity push your personal space boundaries in different ways - both end up making you (or me at least) a bit uncomfortable.
\nSpeaking of uncomfortable - this museum setting feels a bit eerie, but also a bit curious. I found myself wanting to look around to see the environment. This last for about a second. Then of course, a T-Rex wanders in.
\n![\"dino2\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2016/05/dino2.png\")
It's a bit interesting how different folks experience this one. Some of my friends got frightened of the dinosaur and wanted to remove the headset. For me, from a ways back, it was a tiny bit dread inducing - but I could easily write it off as something that didn't exist and therefore didn't scare me in the least as the experience goes on.
\nI felt more of the same as the T-Rex charges at you. It again, wasn't really scary, just minorly uncomfortable. When the dino roars over your head, it's no longer scary at all for me. I found it a bit fascinating and I looked it up and down to soak it all in. I think in part, it's because the T-Rex has officially done all it can do, and you realize it. There's no anticipation anymore.
\n![\"dino\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2016/05/dino.png\")
My second favorite experience was one where you are placed high above a city on an unguarded steel ledge. Lest you think I was putting on a brave face writing about the T-Rex, THIS experience was very uncomfortable and fear inducing for me.
\n![\"heights\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2016/05/heights.png\")
YES, after several tries, it really no longer affects me. But those first few times DO NOT feel good. I had a similar experience playing a mini golf game on the Vive (where you can be super high on a course with no visible safeties around that would virtually prevent you from virtually falling). It's quite interesting to me that this scares me, but a dinosaur does not. While both are virtual, the dinosaur seems easier to reason away. The experienced scared most everyone equally - with the exception of my wife who I still think lacks a reasonable fear of heights. She was excited to see what would happen if you jump off, and was disappointed that there was no mechanism to do so.
\nEither way, it's fascinating to explore this concept of comfort in VR and the ways it can be invaded or not invaded. Even the Oculus store rates their content with comfort levels, with the highest being "intense".
\nAs much as I enjoyed exploring this concept, my favorite experience was unlike all the others and was not uncomfortable in the least.
\nMy favorite experience was what I can only call...a "Paper Village". It's a very stylized, non realistic, cartoony/paperish town floating in the sky. Various animations play from traffic, to airplanes, to a cute little UFO that beams Paper Village citizens up.
\n![\"mini-town2\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2016/05/mini-town2.png\")
There are a few reasons I like it so much. First, it's very self contained. You're looking inward on a tiny world instead of outward at your new virtual environment. To me this turns some of the assumptions you'd make about VR on its head a bit. You're not in it, you are surrounding it. Its a bit god like in that, and a bit silly.
\nAdding to the silliness, is the whole non-photorealistic nature of it. As with the history of computer graphics to date, nothing is perfectly convincing. Realtime VR, as cool as it is, will be the worst at this (behind 3D games, behind rendered movies). Not even attempting this and creating a cartoonish world really encourages your suspension of disbelief and creates something even more fantastic.
\nI think this suspension of disbelief makes you want to peer in every window and explore....
\n![\"mini-town2\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2016/05/mini-town21.png\")
This suspension of disbelief, and this somewhat fantastical environment engages in a way I can't really explain. On top of that it makes me do something that's somewhat embarrassing because I do it every time I'm here even though I fully know the outcome. That something is to reach out and try to touch it. Even though I know it's fake, and even though I can't even see my hands.
\nIts freaky.
\nThis is a good place to end things in this post, because my next post takes this cartoonish experience and explores it in the first piece of VR content I bought.
\n", "date_published": "2016-05-10T00:00:00Z" } , { "id": "https://benfarrell.com/blog/2016-05-10-hands-on-with-the-oculus-rift/", "url": "https://benfarrell.com/blog/2016-05-10-hands-on-with-the-oculus-rift/", "title": "Hands on with the Oculus Rift", "content_html": "Virtual Reality is very fascinating to me. For one, its both a brand new technology given our current state of hardware that makes it possible AND also old as heck. I remember playing "Dactyl Nightmare" back before I graduated high school 20 years ago.
\nhttps://www.youtube.com/watch?v=L60wgPuuDpE
\nIt's now 2016, and the difference between then and now is that we seem poised to get VR in the hands of everyone. Both high end VR experiences because of powerful graphics cards and computers, and lower end VR experiences because everyone has a cellphone and Google Cardboard is virtually free.
\nBesides the coolness factor, I'm interested in VR because early adopters and developers have the opportunity to invent a medium, but more specifically for user experience nerds like myself, we have the opportunity to re-invent many user interface paradigms. We can even question the whole notion of traditional user interfaces in general.
\nBefore we get deep into using VR, I want to cover one possible way consumers have to experience it right now. There are a few.
\nThe HTC Vive is one and offers a pretty stellar experience. It's inclusion of hand tracking allows users to reach out and touch virtual objects. You may be surprised at how much lack of seeing your own body detracts from the experience. When you look down and don't see your chest and feet or reach out to try to touch something and don't see your hands it feels a bit uneasy.
\nThe Vive also offers a bigger area to roam around. There's quite a bit to set up here - but once you put the positional trackers in the corners of your room, VR experiences can track your body moving through a wide area giving you more freedom to virtually move around.
\nAdditionally (and apparently not well advertised because a co-worker just told me), the Vive has a front facing camera which can create a form of Augmented Reality (although future AR promises to be way cooler than having to look through a camera to see the real world).
\nOther than the Vive, the major player are the Oculus Rift, the Gear VR, and Google Cardboard. I won't get into the latter two, because I unfortunately haven't had the pleasure of using the Gear, and the Cardboard...well, it's nifty, not the best, and you can easily use it today with really no cost if you have a smartphone. I highly recommend trying it, but don't think for a second Cardboard is as good as VR gets.
\nI was pretty psyched for VR back when the Oculus Rift was a Kickstarter campaign, before Facebook bought them. So, I backed both the development kits (the DK1 and DK2). Imagine my surprise when I found out that I'd get the Consumer version (CV1) for free!
\nWell, its been delivered and shipped and setup at my apartment. One major problem I had was that my slim Yoga 2 Windows laptop didn't even come close to the minimum specs noted on the Rift. So my first task was to build a desktop gaming computer for myself. I spent around $820 to build it out - meeting (not really exceeding) their specs. I'll put my parts list at the end of this post. This is all on top of the $600 most people will have to pay for the Rift itself. Unfortunately, its certainly not a casual purchase yet. The price, coupled with the fact that people won't see the value without being able to experience it first means that adoption might be a bit difficult.
\nSo with a fresh pair of eyes, before we get deep into specific games and experiences. Let me show you what you get when you buy the Oculus Rift CV1.
\nFirst, the headset:
\n![\"IMG_20160508_192823\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2016/05/IMG_20160508_192823.jpg\")
The headset is the main piece of equipment you'd expect for VR. There's a few more pieces I'll get into after talking about the headset. I don't want to get too far into specs and numbers that you can lookup elsewhere, but I do want to mention a few things.
\nFirst, the screen door effect. It doesn't seem to exist. That's awesome! The so called screen door effect can be seen in older versions of the Rift. In the DK1 and 2, it used to be that you could see the black grid between the pixels because the screen was so close to your face. It was almost as if you were looking through a screen door to see VR. To be fair, some have said you can still see the effect on the new Rift if you squint hard enough - but honestly, I think the Rift's new resolution is pretty awesome.
\nAnother critique of the older Rift was that headtracking could be slow. When your head movement doesn't match the rate of the 3D scene movement, this can be disconcerting even if off by a miniscule amount. It commonly leads to feeling a bit nauseous. To be honest, I never really had that problem with the DK2. It felt fine for the most part, and when it didn't, it was most likely a sub par application. Of course with the new Rift, headtracking STILL seems perfect to me, and also in every experience. This may be due to Oculus firming up their SDK for release. When everyone runs the same code and everyone uses the same head tracking, its probably more consistent.
\nAnother nice attention to detail is the head sensor on the rift. When you put the Rift on, it turns on automatically (likewise when you take it off). Here's an inside shot of the Rift - the sensor is centered above the lenses.
\n![\"IMG_20160508_192837\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2016/05/IMG_20160508_192837.jpg\")
The new Rift for me has all the promise of the old Rift, and none of the more prominent pitfalls. Of course that's a big statement. As we get more used to VR and what it offers we'll most likely see problems with any given platform that we never anticipated - and "no pitfalls" will turn into "many pitfalls".
\nOne of those pitfalls on the horizon is input. Seeing and experiencing VR is great and all, but we need to interact with our virtual world. How we do this is a main differentiating feature of the major players. The Vive's world is room scale, which means you can roam around a pre-defined space much larger than the not-much-more-than-standing space the Rift affords you. The Vive also offers hand tracking which means you can ever so clumsily use your real hands (each gripping a controller) in the virtual world.
\nThe Rift is rumored to offer both of these features soon, but what does it have now? Well here it is...all the extra stuff shipping with it (OK fine it comes with a lens cleaner and handy carrying case too, not pictured):
\n![\"IMG_20160508_192712\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2016/05/IMG_20160508_192712.jpg\")
So, this little guy you'd mount on your desk or somewhere reasonably high and point it at where you intend to use your Rift. It's camera-ish....meaning I don't know if it's technically a real RGB camera, whether it uses depth sensing, if it's simple infrared, or really what it is. But it certainly does track your body movement well. This means when you lean or move around in your small space, your movements are reflected in the virtual world. It sounds like a small thing, but the older Rift didn't have it, and your ability to move felt very rigid and limited.
\n![\"IMG_20160508_192538](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2016/05/IMG_20160508_192538-1.jpg\")
Two input devices ship with the Rift. The controller is what it appears - an XBOX controller. It almost feels like a copout! But I guess until we get imaginative enough with our VR experiences (like the Vive is starting to do), we'll have to settle for treating VR content like our old school consoles for now.
\nPictured to the left is a simpler controller. Its most likely powered by a watch battery and I can't really tell how it connects - I guess it's probably Bluetooth? Either way, its a simple remote. Most of the experiences I've seen (that don't rely on the XBOX controller) don't use more than the central button that you'd press with your thumb.
\nAnd that brings us to actually using the thing. What kinds of experiences does it offer? Well, for absolute starters, you can check out the "Dream Deck" These are example short experiences that give you a good feel for what's possible. To navigate through them, you basically point your face (with a dot for a cursor) at the menu item you want and press the main button on your clicker.
\nSound archaic? Yeah, it kind of is - but again: room to improve and invent! This post is getting a bit long, so I'll cover the Dream Deck in another one.
\nAlso, as promised, here's my parts list for the Windows desktop I built to accommodate this. My disclaimer is that I've never really cared to build a gaming machine - so I don't know how mine compares to any other gamer's machine. I also know that the parts I bought meet the minimum of what Oculus recommends. I ALSO bought them as a package from newegg.com because buying separate components that may or may not work together stressed me out.
\nCase: Fractal Design Core 1000 USB 3.0 Cases FD-CA-CORE-1000-USB3-BL
Optical Drive: LITE-ON DVD Burner SATA Model iHAS124-14 - OEM
Motherboard: GIGABYTE GA-H110M-A (rev. 1.0) LGA 1151 Intel H110 HDMI SATA 6Gb/s USB 3.0 Micro ATX Intel Motherboard
RAM:
\n\nOS: Microsoft Windows 10 Home - 64-bit - OEM
Power Supply: EVGA 100-W1-500-KR 500W ATX12V / EPS12V 80 PLUS Certified Active PFC Continuous Power Supply Intel 4th Gen CPU ...
SSD: ADATA Premier SP550 2.5" 240GB SATA III TLC Internal Solid State Drive (SSD) ASP550SS3-240GM-C
GPU: ZOTAC GeForce GTX 970 4GB
... and other platformers.
\nIn my last post, I was really psyched over the suspension of disbelief, cartoony, fantasy world like experience. It's fitting that my first purchased content would be this game.
\nTo be honest, I bought this for two reasons.
\nThe result was that I was blown away. And this is odd, because if it was a normal 3D game release, it would be INCREDIBLY underwhelming. Even a bit underwhelming for $4.99.
\nWhy? Well, gameplay won't last more than an hour or so. Maybe two. The enemies aren't that good (you're mostly fighting sandwiches that don't do much). The story isn't deep at all, and the graphics are "meh".
\nYou too, reading this post can be pretty ambivalent by looking at a screen capture
\n![\"adventure-time\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2016/05/adventure-time.png\")
Like I said, the graphics are "meh". But allow me to say the first good thing about it, and its that the graphics it does have captures the cartoony nature of the show pretty well.
\nIn VR though? Wow.
\nIn a year or two, I think this game will be as underwhelming in VR as it appears. But props to Turbo Button for making you feel like a part of the game. Right off the bat, it's just crazy cute to live virtually inside this admittedly sparse world and seeing Finn the Human and Jake the Dog interacting with you.
\nAlso right away, the story is very cleverly set up for the medium. I do think content creators should take note of the way you're ingrained into the story and to have that become a mechanism for playing the game.
\nLemme explain...
\nThe game starts with you approaching Finn and Jake on a field as a tiny person/thing/whatever. You're instantly accepted as buds with them. You never see who/what you are because it's all first person view. Unfortunately, Magic Man pops in randomly from out of nowhere and starts wreaking havoc (sounds weird, but actually very in character with the show). Magic Man uses his magic and makes you, the player, incredibly huge. This story mechanism, effectively turns you into the camera.
\nFinn and Jake plod on as you control Finn with your XBOX controller. But you as....well, huge you/game camera both follow them around in hopes Magic Man can be found and subdued into turning you back to normal size. The perspective/size change alone, is something very interesting and ripe to explore in VR. This game only touches it briefly as it's story intro, but all the same, I'd love to see more in other experiences.
\nNow that you're the game camera, very interesting things can be found and, well NOT found.
\nGo back to the first set of popular 3D platformers. Say....Mario 64:
\n![\"mario\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2016/05/mario.jpg\")
Because it was 3D, there was a camera. The camera would awkwardly follow you around, and when it was exceptionally awkward, you'd use your joystick to move it.
\nWith Adventure Time, the camera still follows Finn around...but only loosely. Remember that you are the camera, and peeling the onion skin back, you're wearing a headset on your face that you control as naturally as you would looking around in real life.
\nThe game doesn't have very intricate levels, but there are some hiddenish paths to explore. Free movement of your head as well as the ability to physically lean, duck, stand on your tiptoes in real life adds a VERY interesting element to the old 3D platformer. In some ways I can liken it to controlling a character within a dollhouse in good old fashioned meat space. Its a very unique perspective. I only wish that there were other ways to control it besides the XBOX controller, because that feel in your hands pulls you back to thinking it's fake again.
\nIts so hard to get this point across without experiencing it for yourself. Just imagine being able to standup and look around this environment while your character hangs tight
\n![\"adventure-time2\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2016/05/adventure-time2.png\")
In a further nod to keeping you part of the game, both Finn and Jake will interact with you and talk to you regularly. Sometimes exploiting the infamous cheesy "I'm watching a 3D movie gag" by throwing something in your face. But yah, here's Finn chatting you up:
\n![\"adventure-time3\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2016/05/adventure-time3.png\")
All in all, its so worth $4.99. Probably not worth an extra zero, but I'm really glad I purchased this one as my first VR game. The original voices and sticking to an albeit simplish Adventure Time plot with very Finn and Jake-ish dialog makes me smile.
\nI should also toss a nod to a game called "Lucky's Tale". This game comes free with the Rift, but I didn't try it until after Adventure Time. It's obviously more geared towards kids, as adult me didn't care about the story. It was also a bit boring and cheap just capturing coins as I plod through the levels. Use of the camera in this 3D platformer has the same gameplay mechanic as Adventure Time did, but without getting written into the story. I think by the time I got to Lucky's Tale, my awe and wonder for the re-invention of the 3D platformer was used up, so this fell flat for me. That said, if you're shy about trying it and you have a Rift, it certainly won't cost you anything! And also to be fair, I do think the art direction, style, and level design surpasses Adventure time by a fair bit.
\n![\"luckys-tale\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2016/05/luckys-tale.png\")
What to cover next? I just recently bought Subnautica and the Climb. Both are pretty fascinating, and I'll write these up later. As you can tell, I'm not so concerned with telling you about core gameplay or how fun it is. I even thought I might just analyze user interaction in VR - how it's done in this brave new world, but it turns out that what kinds of feelings this content evokes is a major part of the user experience.
\n", "date_published": "2016-05-10T00:00:00Z" } , { "id": "https://benfarrell.com/blog/2016-02-26-progressive-web-apps/", "url": "https://benfarrell.com/blog/2016-02-26-progressive-web-apps/", "title": "Progressive Web Apps", "content_html": "Last Friday, the SFHTML5 Meetup group met to discuss something called "Progressive Web Apps." I had some preconceived notions that the topic would be pretty cool, but actually, it got me more excited about the state of mobile/web/desktop in 2016 than I could have imagined.
\nThis might sound a bit dramatic, especially given the negative tone that Alex Russel (@slightlylate), the speaker, started off with on the mobile web. Despite being negative, he was spot on, and the talk was a real eye-opener for us who have been working on the mobile web for so long that we forget how much it sucks.
\nAnd yes, it does suck. A good point was made that every other mobile platform started out mobile. No vendor has ever really proposed, "OK, let's take this UI platform, along with everything that's ever been built with it that works on the desktop with mouse and keyboard, and dump it on mobile." Nobody did that until it came to web browsers. It's amazing that things work as well as they do right now.
\nAlex then took us through an exercise of asking for hands up on who used a web-based email client on the desktop. Around 95% of our hands raised. When the question was reversed, "Who uses a web-based email client on their mobile device?" the result was exactly the opposite.
\nWhy does the mobile web suck so much? The folks that have given "Responsive Web Design" (RWD) a shot can't be blamed for this problem. The rest of the web community...if you want your stuff to work on mobile, it's time for a redesign.
\nEven with RWD, some mobile redesign love, and the MOBILE FIRST! mantras we shout, the fundamental user experience with the mobile web, as it is now, will never compete with those for mobile apps. It's probably not because HTML/JS/CSS is slow. Yeah, native can be faster, but if you think about it, most apps you use really don't need speed. If you don't agree with me, tell that to all the app developers using Phonegap, Cordova, or even just a plain WebView for their entire products.
\nSo speed isn't the issue for most apps. Touch, screen orientation, and size don't need to be an issue if the web team cares enough to address them. No, to compete with your typical mobile app, design comes down to how the browser itself runs and loads the page.
\nReal, installed apps have two pretty big advantages over mobile apps:
\nThere's a 3rd advantage, and that's push notifications: messages from the app that appear in the notification area even when the app isn't running. I think that functionality is big-ish, but unless you have a significant base of users addicted to your app (think Facebook), it isn't as big of a deal. Smaller guys and gals are just trying to develop a neat app.
\nProgressive Web Apps attempt to solve all of that missing functionality, and they do so in a way that doesn't necessarily interfere with the current way we develop for the web.
\nTackling the first issue of putting your page on the mobile home screen is pretty important. How the application is displayed, both on the home screen and when it loads, is part of that experience. To solve it, use the "Web App Manifest"! It's a JSON file linked from your HTML head that allows you to define things like app icons, fullscreen display, splash screen, and more.
\nThis is the point when I should confess that I haven't worked with Progressive Web Apps yet. Luckily for me, this isn't a "how-to" article. So for great details on how-to do this stuff, run an easy search, or for your convenience, read this nice technical article via MobiForge.
\nEither way, the idea is that if a user visits your page often enough within a certain time frame, the browser will ask the user if he or she would like to place the page on their home screen. Or, the user can simply add it to the home screen from the options menu in the mobile browser. That's light-years better than having to open the browser, remember the URL, and load the page. I'm sure it's a huge reason apps are winning on mobile right now.
\nSecondly, we have "Service Workers." They sound nerdy and boring, and maybe they are, but the potential they open up for appifying a webpage is huge. Basically, you'd use a Service Worker to intercept a specific set of resources as the webpage fetches them. Yes, if the user is offline that first time they want to access the page, they're outta luck. However, when they do intercept these resources with a connection, they'll be cached. You, the developer, control which files get cached via a Javascript array in the code. On subsequent loads, even if the user is offline, the page can load with your cached assets, whether they are images, Javascript, JSON, styles, or whatever. Here's a better technical description of how that works.
\nIn fact, Google has published documentation and some tools on the similar notion of an "Application Shell Architecture" wherein persistent assets that don't change can be cached, but dynamic content that isn't cached will update.
\nProbably the most exciting thing about Progressive Web Apps is that both the Manifest and the Service Workers will not affect the web page negatively if the browser doesn't support the features. This means that the worst you can do is waste time and JS code on something that doesn't pan out as you hoped.
\nAnd there is some danger that it won't work. You may have noticed that Facebook today uses push notifications with Service Workers and that they do increase engagement on their site. So that's a win! Unfortunately, Service Workers and the Web App Manifest aren't supported everywhere. Unsurprisingly, that means they're pretty much everywhere but iOS/Safari. Even worse, browser vendors on iOS can't use their own web engines to support the Progressive Web Apps -- under the hood, both Chrome and Firefox have to use Safari tech.
\nApple seems tight-lipped about whether they intend to adopt Progressive Web Apps at all. I'm going to say that for now, it doesn't matter. If you've hung around the SF Bay Area enough, you may have noticed that many companies operate on an "iOS first, Android distant second" agenda. That doesn't make sense in that Android devices far surpass iOS devices in sales. But it does make sense, in that iOS app sales are greater and it can be daunting to develop apps for the large ecosystem of Android devices on the market.
\nHowever you slice it, Android is second for developers, which is bad for consumers. Right now, many companies will adopt a Web + iOS + maybe Android strategy. If they can combine the Android + Web strategy with Progressive Web Apps AND not force folks through the Google App Store, it'll be a huge win for everyone. I'm guessing Google probably doesn't even care much about having an app store, save for the fact that it was necessary to maintain a mobile ecosystem.
\nMeanwhile, the point was made at this Meetup that with every additional step a user must go through to download an app, there's around a 20% dropoff rate. Think about how many steps there are in clicking an app link, going to the store, starting the download, waiting for the install, and finally opening the app -- many apps are losing out on users. And let's face it, the app gold rush is over. There are some lottery winners still, but most apps are too costly to make and market to justify what they bring in return.
\nProgressive Web Apps short circuit that whole process by eliminating app discovery and install. While Android users will enjoy a huge user experience win, Apple will most likely try to maintain their stranglehold on their app store and come kicking and screaming only once web devs demand these new features.
\nWhat's more, and what I'm really excited for, is our return to disposable digital experiences. Hate Adobe Flash or not, it really created a heyday for disposable experiences: Flash games to play a couple times and get bored with, nifty digital playgrounds, etc. It's way harder to convince someone to download an app than it is to go to a webpage and pop it on their homescreen until they get bored of it in a week.
\nTo extend, I think Progressive Web Apps will also be a huge boon for web-based Virtual Reality. Immersive experiences will come from many different places and frankly, will not be wanted as a permanent app install. Already, we're seeing the rise of VR portals like MilkVR because smaller, one-off VR experiences need some kind of entry onto a device. When Progressive Web Apps make WebVR easier to get before eyes than an app portal, VR will win big.
\nTo reiterate, I think Progressive Web Apps are the next big thing for mobile, potentially replace lots of simple apps, and will mark the return of fun, disposable experiences. I don't have the technical experience with these new tools to back me up yet, but I will soon. Don't take my word for it, though! Read up on it and try it yourself.
\nHere's another post from the aforementioned Alex Russel: https://infrequently.org/2015/06/progressive-apps-escaping-tabs-without-losing-our-soul/
\n", "date_published": "2016-02-26T00:00:00Z" } , { "id": "https://benfarrell.com/blog/2015-10-27-es6-web-components-part-5-wrapup/", "url": "https://benfarrell.com/blog/2015-10-27-es6-web-components-part-5-wrapup/", "title": "ES6 Web Components Part 5 - Wrap-Up", "content_html": "In Part 4 of my 5-part write-up, Project Setup and Opinions, I talked about lessons I took away from experimenting with ES6 Web Components. Lastly, is my wrap-up post...
\nThis was a monster write-up! In my four previous parts, I've shown you the basics on Web Components, what features make up a Web Component, how ES6 can help, and some coding conventions I've stumbled on through my experimentation.
\nThat last sentence is my big caveat - it's trial and error for me. I'm constantly experimenting and improving my workflow where it needs to be improved. Some pieces I've presented here, but I may come up with an even better way. Or worse, I may discover I showed you folks a really bad way to do something.
\nOne particular thing to be cautious of is recognizing I'm not talking about cross-browser compatibility here. I have done a bit of research to show that, theoretically, things should work cross-browser, especially if you use the WebComponents.js polyfill. I have done a little testing in Firefox, but that's it. I really haven't tested in IE, Edge, Safari, et cetera. I'm lucky enough to be in a position right now at my job and in my personal experiments where I'm focusing on building in Chrome, Chromium, or Electron (built on Chromium). I'm trying to keep compatibility in mind; however, without a real effort to test in various browsers, you may run into issues I haven't encountered.
\nIt isn't all doom and gloom, though. WebComponents.js is used as the Google Polymer polyfill. Its why Polymer claims to have the cross-platform reach it has. See the support grid here for supported browsers.
\nEven better, as I complete this series, Webkit has just announced support for the Shadow DOM. This is fantastic, because the Shadow DOM is the hardest piece to polyfill. A while back, Polymer/WebComponents.js had removed polyfilled Shadow DOM support for CSS because it wasn't very performant. Microsoft announced a while back that it's working on the Shadow DOM, while Firefox has it hidden behind a flag.
\nAll this is to say, if you take anything away from this series of blog posts on ES6 Web Components, takeaway ideas. Treat them as such. Don't take this to your team and say "Ben Farrell has solved it all; we're all in on Web Components." I truly hope everything I've said is accurate and a fantastic idea for you to implement, but don't risk your production project on it.
\nWith all that said, aside from the implementation details, I do think Web Components are a huge leap forward in web development. It's been encouraging me to use pure vanilla Javascript everywhere. I haven't needed jQuery, syntactic sugar provided by a framework, nontraditional markup for binding - it's all pure JS. I have been using JS techniques like addEventListener, querySelector, cloneNode, et cetera. Those are all core JS, CSS, and HTML concepts. When you understand them, you understand what every JS framework and library is built on. They transcend Angular, React, jQuery, Polymer, everything. They will help you learn why your favorite tool is built the way it is and why it has the shortcomings it does.
\nNot only am I building pure JS here, but I'm organizing my code into reusable and modular components - what every JS framework tries to give you.
\nFor these reasons, I think there is huge potential in Web Components and I think it most likely represents what we'll be doing as a community years from now, especially when (hopefully not if) all features of Web Components and ES6 are implemented in browsers everywhere.
\nAs I said in my first post, I do like Google's Polymer a lot. But again, I strive to do less application-like things and more creative-like things. Therefore, MY components are fairly custom and don't need a library of Google's Material-designed elements. I've started a Github Org called Creative Code Web Components that contains a video player and camera that draw to the canvas and effects can be created for them on the pixels. I've created a speech-input component as well, along with a pure ES6 Web Component slide deck viewer.
\nThose components are all in early stages, but for fabricating various creative projects, I feel like this the right way forward for me. Thus far, I have a real modular set of pieces for creating a neat prototype or project.
\nPerhaps if you are doing a real business application, Polymer is great for you. Or React. Or Angular. Regardless, I think what I've been learning is great info for anyone in web dev today to have. I wouldn't have written 10,000 words about it otherwise!
\nThis has been my big 5-part post about creating Web Components with ES6. To view the entire thing, check out my first article.
\n", "date_published": "2015-10-27T00:00:00Z" } , { "id": "https://benfarrell.com/blog/2015-10-27-es6-web-components-part-4-project-setup-and-opinions/", "url": "https://benfarrell.com/blog/2015-10-27-es6-web-components-part-4-project-setup-and-opinions/", "title": "ES6 Web Components Part 4 - Project Setup and Opinions", "content_html": "This article continues my ES6 Web Components series. The last article was the third in the series: Making an ES6 Component Class.
\nSo far, the basics have been pretty....basic. I hope I've given some ideas on how to create ES6 Web Components - but these basics only go so far. I do have some opinions on how to take this further, but they are only opinions that have made sense to me. The beauty of this is that you can hear me out and decide for yourself if these ideas are good for you.
\nLets start with dependencies. I like Babel to compile the ES6 and Gulp to do the tasks. Source maps are also a must in my book for debugging the compiled ES6 as Javascript! Also, given that WebComponents.js has been so instrumental in providing cross platform functionality, lets use that too.
\nHere's a sample package.json file:
\n![\"\"](\"http://ir-na.amazon-adsystem.com/e/ir?t=thegou07-20&l=as2&o=1&a=B00CUSUV0O\")
{\n "name": "ccwc-slideshow",\n "version": "0.1.0",\n "dependencies": {\n "webcomponents.js": "^0.7.2"\n },\n "devDependencies": {\n "babel": "^5.8.21",\n "gulp": "^3.9.0",\n "gulp-babel": "^5.2.0",\n "gulp-sourcemaps": "^1.5.2"\n }\n}\nNext up is Gulp. I have nothing against Grunt...but I use Gulp. Frankly I stopped caring about the battle of the task runners and landed on the last one that I tried and liked. There probably won't be too many tasks - I just need to compile the ES6 to Javascript. However, I may have multiple components in my repo. As a result, I'll have a compile task per component. Here's a sample Gulpfile:
\nvar gulp = require('gulp');\nvar sourcemaps = require('gulp-sourcemaps');\nvar babel = require('gulp-babel');\nvar concat = require('gulp-concat');\n\ngulp.task('slide', function () {\n return gulp.src('src/ccwc-slide.es6')\n .pipe(sourcemaps.init())\n .pipe(babel())\n .pipe(concat('ccwc-slide.js'))\n .pipe(sourcemaps.write('.'))\n .pipe(gulp.dest('./src'));\n});\n\ngulp.task('slideshow', function () {\n return gulp.src('src/ccwc-slideshow.es6')\n .pipe(sourcemaps.init())\n .pipe(babel())\n .pipe(concat('ccwc-slideshow.js'))\n .pipe(sourcemaps.write('.'))\n .pipe(gulp.dest('./src'));\n});\n\ngulp.task('default', ['slide', 'slideshow']);\nOne last Javascript note: I like to have an ES6 extension on my ES6 files, and those to live in a "src" older. Some folks seem to be using .js, and then compiling them to "mycomponent-compiled.js". I don't like this for a couple of reasons. First, its not obvious that your source JS file is ES6, and secondly I kinda think it's silly to force devs to use a non-obvious naming convention when including a script. When you make your script tag, you should link to "mycomponent.js". Not "mycomponent dot, ummm...what was my naming convention last week?".
\nYour Web Component HTML files should live in your project root. When you link to a Web Component, you shouldn't need to remember what folder you put your stuff in...it should be a simple and easy to remember "mycomponent/mycomponent.html".
\nLastly, your demo is important! A Web Component should demonstrate use! When I started out, I was making a "demo" folder in my component root, and putting an index.html or demo.html file in there. There's a problem with this though: if you use images (or other assets), the relative path to your image will be different from the demo folder than what it is during actual use of your component. Bummer. So I like to put a "demo.html" usage example in my component root. I still have a demo folder, but this folder would contain any assets to support running the demo that aren't really part of your component (like JSON data).
\nActually - one more. This is the last one for real. Documentation for your component. I didn't think about it here, because I didn't even think of doing it for my components yet. My bad. My horrible horrible bad. Google's Polymer actually has a very nice self documenting structure which is very sweet. Maybe someday, I'll base whatever I plan to do about docs on that.
\nHere's a sample project structure of a component I made for showing a slide deck. You'll notice 2 components here. One is the slide deck viewer, and one is a component to show a single slide. The first uses the second inside it and it all works together. I have some sample slide deck contents in my demo folder:
\n![\"componentstructure\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2015/10/componentstructure.png\")
You'll notice that I have the compiled .js.map files and the .js files here, too. I check these in to source control. I always feel a little icky about checking in compiled files. For one, they don't NEED to be in source control since they are generated and don't need to be diffed. Secondly, you don't want to allow people to edit these files instead of editing the ES6 files. Lastly, I am occasionally forgetful of building then checking in! Sometimes only the ES6 files get checked in, and I'm left doing another commit of the compiled files when I remember that I didn't build.
All that said, I DO check these compiled files in. For my workflow, I want these file instantly useable after NPM installing my component. Forcing a compile step for an NPM module and requiring dev dependencies seem like an unnecessary burden on the end user. I'm always trying to think of ideas to make myself happy here on all counts, but I haven't yet.
\nI've already documented the bare minimum of methods in your Web Component class. These include: "attachedCallback", "createdCallback", "detachedCallback", and "attributeChangedCallback". These, however, are just HTMLElement lifecycle callbacks. I have some other methods of my own I like to consistently use (all invented by me, of course, and not part of any spec).
\nsetProperties
\nIn ES6, there is no attaching properties directly to the class inline. Properties can only be set on the class from within a method. So I made my own convention that I consistently use. My "setProperties" method initializes all of the variables that would be used for the class. In other languages, public/private variables would be used at the top of a class. In ES6, I use my "setProperties" method for this and give my variables a little extra documentation/commenting.
\nparseAttributes
\nOnce the component is created or attached, you may want to look at the attributes on your component tag to read in some initial component parameters. You could potentially scatter these all over your code. I like to read them all in one spot: "parseAttributes".
\nregisterElements and the "dom" object
\nI really dug Polymer's "$" syntax. Anything in your component's HTML that had an ID, you could access with "this.$.myelement". Well, in my DIY Web Component world, I can't just magically expect to access this. I COULD querySelector('#myelement') everytime, but its more performant to save these references to a variable if you're using them often. And it also creates more readable code to save your important element references in well named variables. At the same time, though, it might be confusing to mix elements on your root "this" scope with other variables that aren't elements.
\nSo here's what I do...
\nWhen I have a bunch of stuff that I want to reference in the imported HTML template at the very start, like buttons, text fields, whatever, I'll run my custom method "registerElements" in the attachedCallback after appending the template to my Shadow Root.
\nIn "registerElements", I'll create a new Object called "dom" on my root scope "this" (this.dom = {};). I'll then querySelect any elements I want, grab the reference, and populate "this.dom.myelement" with the references. Then elsewhere in my code, I can just reference the property like a normal variable (but I know it's a DOM element since its in my "this.dom" object).
\nroot
\nOne last thing I do consistently....and this is not a method, but a property...is using a custom variable "root" to represent the Shadow DOM. So when I want to use querySelector on an element, I use "this.root.querySelector('myelement')". I COULD just call it "shadow". However, there's been a couple times I've been a bit wishy-washy about using the Shadow DOM, and I can just set "this.root" to the host content, or even the document if I wanted. In this fashion I can keep swapping around what "root" is to whatever I choose and keep my code pretty much the same.
\nI'll leave you with a complete example of my Web Component that functions as a Slide Deck viewer. Remember, the slide inside is a web component on its own! In my next post, I'll wrap this whole thing up and link you all to my real components.
\n\nclass CCWCSlideShow extends HTMLElement {\n setProperties() {\n /**\n * slides deck\n *\n * @property deck\n * @type string\n */\n this.deck = '';\n\n /**\n * next slide key mapping\n *\n * @property nextSlideKey\n * @type integer\n */\n this.nextSlideKey = 39; // right arrow key\n\n /**\n * previous slide key mapping\n *\n * @property previousSlideKey\n * @type integer\n */\n this.previousSlideKey = 37; // left arrow key\n\n /**\n * toggle timer key mapping\n *\n * @property toggleTimerKey\n * @type integer\n */\n this.toggleTimerKey = 84; // "t" key\n\n /**\n * timer start time\n *\n * @property timer start time\n * @type Number\n */\n this.timerStartTime = 0;\n\n /**\n * current slide/chapter\n *\n * @property current slide/chapter\n * @type object\n */\n this.current = { chapter: 0, slide: 0 };\n\n /**\n * running\n * is slide deck running (being timed)\n * @property running\n * @type boolean\n */\n this.running = false;\n\n /**\n * slides\n *\n * @property slides\n * @type array\n */\n this.slides = [];\n };\n\n /**\n * register dom elements\n */\n registerElements() {\n this.dom = {};\n this.dom.slideviewer = this.root.querySelector('#slideviewer');\n this.dom.slideinfo = this.root.querySelector('.infobar .slides');\n this.dom.runtime = this.root.querySelector('.infobar .runtime');\n };\n\n\n /**\n * ready\n *\n * @method ready\n */\n init() {\n this.loadDeck('./deck/manifest.json');\n document.addEventListener('keyup', event => this.onKeyPress(event) );\n\n setInterval( () => {\n if (this.running) {\n var duration = Math.floor((new Date().getTime() - this.timerStartTime) / 1000);\n var totalSeconds = duration;\n var hours = Math.floor(totalSeconds / 3600);\n totalSeconds %= 3600;\n var minutes = Math.floor(totalSeconds / 60);\n var seconds = totalSeconds % 60;\n if (seconds.toString().length == 1) {\n seconds = "0" + seconds;\n }\n if (minutes.toString().length == 1) {\n minutes = "0" + minutes;\n }\n this.dom.runtime.innerText = hours + ":" + minutes + ":" + seconds;\n }\n }, 1000);\n };\n\n /**\n * toggle timer\n *\n * @method toggleTimer\n */\n toggleTimer() {\n this.running = !this.running;\n if (this.timerStartTime === 0) {\n this.timerStartTime = new Date().getTime();\n }\n };\n\n /**\n * on keypress\n * @param event\n */\n onKeyPress(event) {\n switch(event.keyCode) {\n case this.nextSlideKey:\n this.nextSlide();\n break;\n\n case this.previousSlideKey:\n this.previousSlide();\n break;\n\n case this.toggleTimerKey:\n this.toggleTimer();\n break;\n }\n }\n\n /**\n * load chapter in slide deck\n * @param index\n * @param uri\n */\n loadChapter(index, name, uri) {\n var xmlhttp = new XMLHttpRequest();\n xmlhttp.onreadystatechange = () => {\n if (xmlhttp.readyState == 4) {\n if (xmlhttp.status == 200) {\n var chapter = JSON.parse(xmlhttp.responseText);\n chapter.index = index;\n chapter.name = name;\n this.chapters.push(chapter);\n this.chapters.sort(function(a, b) {\n if (a.index > b.index) { return 1; } else { return -1; }\n });\n this.manifest.slideCount += chapter.slides.length;\n this.goSlide(0, 0);\n }\n }\n };\n xmlhttp.open("GET", uri, true);\n xmlhttp.send();\n };\n\n /**\n * load deck\n * @param uri of manifest\n */\n loadDeck(uri) {\n var xmlhttp = new XMLHttpRequest();\n xmlhttp.onreadystatechange = () => {\n if (xmlhttp.readyState == 4) {\n if (xmlhttp.status == 200) {\n this.manifest = JSON.parse(xmlhttp.responseText);\n this.manifest.slideCount = 0;\n this.dom.slideviewer.imgpath = this.manifest.baseImagePath;\n this.dom.slideviewer.htmltemplatepath = this.manifest.baseHTMLTemplatePath;\n\n this.chapters = [];\n for (var c = 0; c = this.chapters[this.current.chapter].slides.length) {\n this.current.slide = 0;\n this.current.chapter ++;\n\n if (this.current.chapter >= this.chapters.length) {\n this.current.chapter = 0;\n }\n }\n this.goSlide(this.current.chapter, this.current.slide);\n };\n\n /**\n * previous slide\n */\n previousSlide() {\n this.current.slide --;\n if (this.current.slide < 0) {\n this.current.chapter --;\n\n if (this.current.chapter < 0) {\n this.current.chapter = this.chapters.length - 1;\n }\n this.current.slide = this.chapters[this.current.chapter].slides.length - 1;\n }\n this.goSlide(this.current.chapter, this.current.slide);\n };\n\n /**\n * go to slide\n * @param {int} index of chapter\n * @param {int} index of slide\n */\n goSlide(chapter, slide) {\n this.current.chapter = chapter;\n this.current.slide = slide;\n\n var slidecount = slide;\n for (var c = 0; c {\n this.dom.slideviewer.setText(item.html, item.region);\n });\n }\n\n if (sld.images) {\n sld.images.forEach( item => {\n this.dom.slideviewer.setImage(item.image, item.region);\n });\n }\n\n if (sld.background) {\n this.dom.slideviewer.setBackgroundImage(sld.background, sld.backgroundProperties);\n }\n };\n\n /**\n * getter for slide element\n *\n * @return slide element\n */\n getSlideComponent(id) {\n return this.dom.slideviewer;\n };\n\n /**\n * getter for slide element\n *\n * @param {string} class name\n * @return {array}\n */\n getHTMLIncludeElementsByClass(clazz) {\n return this.getSlideComponent().getHTMLIncludeElementsByClass(clazz);\n };\n\n // Fires when an instance was removed from the document.\n detachedCallback() {};\n\n // Fires when an attribute was added, removed, or updated.\n attributeChangedCallback(attr, oldVal, newVal) {};\n\n /**\n * parse attributes on element\n */\n parseAttributes() {\n if (this.hasAttribute('deck')) {\n this.deck = this.getAttribute('deck');\n }\n\n if (this.hasAttribute('nextSlideKey')) {\n this.nextSlideKey = parseInt(this.getAttribute('nextSlideKey'));\n }\n\n if (this.hasAttribute('previousSlideKey')) {\n this.previousSlideKey = parseInt(this.getAttribute('previousSlideKey'));\n }\n\n if (this.hasAttribute('toggleTimerKey')) {\n this.toggleTimerKey = parseInt(this.getAttribute('toggleTimerKey'));\n }\n };\n\n\n // Fires when an instance of the element is created.\n createdCallback() {\n this.setProperties();\n this.parseAttributes();\n };\n\n // Fires when an instance was inserted into the document.\n attachedCallback() {\n let template = this.owner.querySelector('template');\n let clone = document.importNode(template.content, true);\n this.root = this.createShadowRoot();\n this.root.appendChild(clone);\n this.registerElements();\n this.init();\n };\n}\n\nif (document.createElement('ccwc-slideshow').constructor !== CCWCSlideShow) {\n CCWCSlideShow.prototype.owner = (document._currentScript || document.currentScript).ownerDocument;\n document.registerElement('ccwc-slideshow', CCWCSlideShow);\n}\n\nContinue on to the conclusion of my ES6 Web Component Series
\n", "date_published": "2015-10-27T00:00:00Z" } , { "id": "https://benfarrell.com/blog/2015-10-27-es6-web-components-part-3-making-an-es6-component-class/", "url": "https://benfarrell.com/blog/2015-10-27-es6-web-components-part-3-making-an-es6-component-class/", "title": "ES6 Web Components Part 3 - Making an ES6 Component Class", "content_html": "One thing I didn't cover in my last post (#2 in the series) about all the pieces that bring Web Components together is actually using ES6! It's not a feature of Web Components, but its some seriously nice glue that lets you tie your Web Component structure together.
\nTo be honest, I'm not using it to it's full capacity. In fact, I'm only using 2 or 3 features to make my life easier!
\nThere are some strong opinions NOT to use classes especially from one prominent person in the JS community. Even if Crockford had all the best intentions, I doubt he had Web Components in mind - because to me, Web Components are the perfect place to use Classes. Components are fairly OO by nature. I want to take an HTMLElement and extend it to my own custom component. One thing I didn't get into when talking about creating my own elements before was the methods that you get when extending HTMLElement. Now is as good a time as any to bring them up:
\n\nclass MyCustomClass extends HTMLElement {\n // Fires when an instance was removed from the document.\n detachedCallback() {};\n\n // Fires when an attribute was added, removed, or updated.\n attributeChangedCallback(attr, oldVal, newVal) {};\n\n // Fires when an instance was inserted into the document.\n attachedCallback() {\n // Remember this? We're cloning the template tag and dropping it\n // onto our page in our newly created Shadow Root\n var template = this.owner.querySelector('template');\n var clone = document.importNode(template.content, true);\n this.root = this.createShadowRoot();\n this.root.appendChild(clone);\n };\n\n // Fires when an instance of the element is created.\n createdCallback() {};\n}\nif (document.createElement('my-customclass').constructor !== MyCustomClass) {\nMyCustomClass.prototype.owner = (document._currentScript || document.currentScript).ownerDocument;\n document.registerElement('my-customclass', MyCustomClass);\n}\nSo the above code are all the methods you get when you extend an HTMLElement AND what an empty ES6 Class might look like. The methods inline with the comments seem pretty self explanatory, but I will mention this...pay VERY close attention to "attachedCallback" vs "createdCallback". Yes, its obvious that one fires when the element is created vs added to the DOM, however - make sure you consider that difference especially when you create an instance of your element at runtime with Javascript. If you did a bunch of cool stuff with "attached", but then did document.createElement('my-customclass') to create your element using JS....it hasn't been added to the DOM yet! So whatever you have in "attachedCallback" hasn't been run yet and your component might not act like you expect!
\nAnd now the non-obvious stuff...that weird "if" statement I have outside of my component. Well, this is all to register your custom element with the browser.
\nFirst, we try to create the custom element. If this is the second time you've used the component, it stands to reason that it's all been registered before and we'd be writing over the original element! So, we create our custom element, and if it's constructor is our class (MyCustomClass), then it's already been created and we shouldn't so it again.
\nNext, I'm using a little known feature: document.currentScript.ownerDocument. Unfortunately, document.currentScript is not present in all browsers. Yet again, WebComponent.js is there to save our bacon. In this scenario, with a polyfill though, _currentScript appears with an underscore, so you have to handle the logic.
\nIn any event, this "ownerDocument" tells us the "document" owned by our script. In this context, document is our component's local DOM! So, super useful, right? If you want to append any children, you do it to this ownerDocument. Create a ShadowDOM? Do it on this ownerDocument. Clone your content? Do it on this ownerDocument. Here, I've taken this "ownerDocument" and assigned it to the class's prototype so we can reference it on any instantiation of the class.
\nAfter all that, we can simply register the tag to the document, providing our custom class for how it should act.
\nIn the end, we have a custom class with a few component lifecycle methods provided for us, as well as a way to create a brand new CUSTOM tag using that class for our custom features.
\nKind of a silly name. But it's easy to remember, because it literally is a fat arrow: => (as opposed to a skinny arrow (->).
\nAnyway, Fat Arrows help you manage scope. Scope can be a huge pain in the butt in Javascript. Before I got into this workflow with ES6 and Web Components, I was using a little trick when you instantiate your JS object, you can create a variable called "self". Since when you first instantiate your object, the variable "this" always refers to the newly instantiated object, by just assigning "var self = this;", you can always refer back to "self" even if you're in a completely different scope because you're off on a click handler or a timer.
\nI'd go into this more, but it's irrelevant. it's all out the window when you assign methods and properties to the underlying Object prototype and not the instance. And that's exactly what ES6 classes do. So how do we get around this, and control our scope?
\nThat's where fat arrows come in to play. By using them, you're passing your outer "this" scope to whatever your arrow points at. Take an event listener for example:
\n\nthis.myawesomeelement.addEventListener('click', event => this.onClick(event) );\nWith this, you can make a method on your class called "onClick" and when it's called, the scope of "this" is STILL the scope of your class. Without the fat arrow, when you tried to refer to "this" in "onClick", it would be in the scope of the event. There would be no way to refer back to your class!
\nTo be honest, I like this a whole lot more than "var self = this" because it always felt like a weird kludge. Yay for ES6! The syntax of the fat arrow can be a little confusing as you try to apply it to different scenarios - expect to look up examples online for your use case as you get used to the syntax.
\nNext up in part 4: Some opinions, workflows, and project setup for ES6 Web Components...
\n", "date_published": "2015-10-27T00:00:00Z" } , { "id": "https://benfarrell.com/blog/2015-10-27-es6-web-components-part-2-the-building-blocks/", "url": "https://benfarrell.com/blog/2015-10-27-es6-web-components-part-2-the-building-blocks/", "title": "ES6 Web Components Part 2 - The Building Blocks", "content_html": "This post is the second of five. Click here to read the first which covers why I came to Web Components in the first place.
\nLet's dig in. In part one, I covered WHY I wanted to write ES6 Web Components, but didn't show any code. Let's change this! Since Web Components are a collection of several seemingly unrelated techniques, bear with me as I cover one at a time and then we'll combine them all.
\nYou're most likely familiar with tons of tags/elements in HTML. You can make a div, a span, an h1, a strong tag, but did you know you make up your own tag?
\nTo make sure I'm not steering you all wrong in my explanations, I went over to HTML5Rocks to get more basic details than I need. I did learn some cool things like the fact that if you create your own tag without a dash in the name, it inherits from HTMLUnknownElement. There's lots of cool facts to drill deeper on there if you want to be a custom tag nerd, but for our purposes, lets just talk about what we need to get up and running.
\nEvery custom element you make needs to have at least one dash in it. I like this limitation, because it's an opportunity to namespace our work. My name is Ben, so if I had a set of elements I was making, I might call a button:
To make my button, I can just do something like this:
\n\ndocument.registerElement('bens-button');\nThat's great (and valid), but Ben's Button doesn't do anything that a DIV tag doesn't do. It extends from something called "HTMLElement", which is just a generic sort of tag.
\nYou can pass in a second parameter here as well. Maybe you don't want it to extend from "HTMLElement", but the "HTMLButtonElement" to make it act more like a button.
\n\ndocument.registerElement('bens-button', HTMLButtonElement);\nYou could even use JS to clone a base element like this, and make it act very custom. That's our ultimate plan, but I'm going to do this with ES6, so lets wait a bit before we dig in.
\nBefore moving on, I should warn you that this is not a Web Standard. It is only officially supported in Chrome and Opera. You can enable it in Firefox with a configuration flag. Luckily, WebComponents.js offers a polyfill for it.
\nStrictly speaking, and depending how far you want to go with Web Components, HTML Imports are fairly optional. Maybe I just need simple markup in my custom element and I'm happy to create it all in Javascript. Maybe I don't care to use CSS either and I can just tweak all my style properties with JS as well.
\nFor more complex custom components OR for the simple purpose of keeping things readable by JS, CSS, and HTML devs alike, I like to use HTML imports. These imports can suck in an HTML file onto your page. The HTML file can include Javascript and CSS to make your component whole.
\nSo for example, in your application you can do this to include the component:
\n\n<link rel="import" href="bens-button.html">\nOf course that's not the whole picture. What's in that HTML file?
\nWell, it could be as simple as just a script reference for Javascript that extends the HTMLButtonElement and does fancy things to it:
\n\n<script src="src/bens-button.js"></script>\nTo go all the way though, we can include HTML and CSS:
\n<script src="bens-button.js"></script>\n<template>\n <span>A Button</span><button>Click Me</button>\n <style>\n :host button {\n width: 100px;\n height: 100px;\n }\n </style>\n</template>\nBefore I break this down, allow me to first say that there are tons of different ways to go about this. What I'm presenting here is a way that I found works great for me. The bad thing about going without a framework is that you have to invent the best way to solve things. Since I'm giving you ONE way to solve things, hopefully this takes most of the badness out of a no framework solution and allows you to explore the GOOD part about not using a framework: You have complete control to tweak my ways and inject your own creative problem solving!
\nSo the first thing we're doing is simply embedding a script tag. This script tag would contain some script to register a new DOM element and customize the hell out of it so that it truly is "bens-button".
\nIn the next line, we have a tag. Templates are cool - they allow us to mark up some HTML, but I doesn't get rendered! It just sits there for us to pull it wherever we want. It would be the equivalent of writing some sort of HTML string in Javascript and use/render it later. But here, it is HTML so we can copy/paste and test elsewhere.
\nHow I write the CSS has been an ongoing battle, and a little bit fuzzy in my mind. First off, I've not been successful at including external CSS files. Unless the CSS is directly included in the template inline, I've been running into issues where it works in some browsers but not others - very much likely due to my use of the Shadow DOM which we'll cover next. If I remember correctly, linked stylesheets might work in Firefox, because it doesn't support the Shadow DOM, so styles can reach into anywhere. In Chrome where I DO use the Shadow DOM, styles cannot reach inside a component, and it looks like linking a stylesheet might be considered "piercing the Shadow DOM".
\nEither way, I'm pretty fuzzy on the logistics here, but again - the above example works for me. The other interesting thing you might notice here, is ":host". I think the most concise explanation of :host comes again from HTML5Rocks: "The :host allows you to select and style the element hosting a shadow tree".
So since I plan to use the Shadow DOM feature, ":host" allows me to target the component's world that I'm working with. Got a DIV with a class of ".mydiv" in my component's Shadow DOM? Well target it with
\n :host .mydiv {\n...\n}\n":host" is pretty useful, but so would "::shadow" or "/deep/" if they weren't being deprecated. "/deep/" allows you to pierce into another component and change style properties. "::shadow" seems a lot like ":host". But again: deprecated, I can't recommend using them.
\nHope that makes sense! To cap this all off, remember I said that doesn't actually render anything? It just sits there in your DOM? Well, let me show you a snippet of Javascript to actually pull it in:
\n\nvar template = mydocument.querySelector('template');\nvar clone = document.importNode(template.content, true);\nelementtoppendtemplateto.appendChild(clone);\nIn the above code, I'm skirting around a couple things. What is "mydocument"? Well, it could be a bunch of things depending on if you are actually using Web Components, the Shadow DOM, or if you are introducing any other nuances in your custom work. I promise to get back to this in part 3 when I tie it all together. Same with "elementtoappendtemplateto". Either way, the gist is that we're first selecting the template from our HTML import, then cloning it. You may ask why I didn't use "cloneNode" instead. For the life of me, I can't remember. I remember that cloneNode didn't work in this context for a very subtle reason (possibly cross browser or because of the ShadowDOM or because the node isn't active in this template), and that importNode is the thing that saved the day. Either way, once I have a clone, I simply append that clone as a child to wherever I want.
\nAnd last but not least...the thing that EVERYBODY thinks of when talking about Web Components is:
\nIronically, the Shadow DOM is probably the least important aspect of Web Components. Don't get me wrong, I think it's brilliant - you get to create a shielded space where your CSS and Javascript can't reach outside your component except in very controlled ways. Other folks can't reach in with their crazy jQuery selectors and accidentally change stuff inside the component.
\nUnfortunately, though, we're a ways off from this awesome panacea. Chrome and Opera are the only ones supporting the Shadow DOM. Firefox only supports it behind a configuration flag, and Safari had it, but then pulled it out, but just this week Webkit announced support.
\nLuckily our webcomponents.js polyfill allows us to use the Shadow DOM. Unfortunately, due to performance reasons, using the polyfill doesn't provide your CSS with a Shadow DOM! So your JS might be protected, but CSS properties can still bleed through in browser that don't support the Shadow DOM. In the end, it's best not to rely on the Shadow DOM right now for style unless you are exclusively targeting Chrome. Tread carefully with this! To be honest, I haven't done my due diligence yet to see how this impacts a larger project and have no horror stories yet - though I'm sure I will have them!
\nIf that scares you enough...feel free to just opt out of the Shadow DOM! Mozilla's X-Tags project has a great Web Components project going.
\nIts fairly easy to opt in OR opt out. It's simply another layer of DOM that you're adding on between your component and the outside world:
\n\nmyroot = this.createShadowRoot();\nAfter that point, any elements you add, including templates (like we talked about above) can be appended directly onto that "myroot" var. We'll call "myroot" something else soon when we string all of these concepts together.
\nSo there are the basic concepts of Web Components! Lots of folks including HTML5Rocks go into some good detail as well, so if you didn't understand it here, or simply want more info - there is tons of info you can find!
\nUp next in part #3, I'll be taking all of these concepts and bringing them together under ES6!
\n", "date_published": "2015-10-27T00:00:00Z" } , { "id": "https://benfarrell.com/blog/2015-10-27-es6-web-components-part-1-a-man-without-a-framework/", "url": "https://benfarrell.com/blog/2015-10-27-es6-web-components-part-1-a-man-without-a-framework/", "title": "ES6 Web Components Part 1 - A Man Without a Framework", "content_html": "Before I launch into this 5-part series of posts, I just want to give a high-level overview of it: I wrote a lot. Partially because I'm long overdue on a blog post on all the stuff I've been experimenting with for the past several months, but mostly because much of the web tech I've been looking into is Web Components and I'm truly excited about it and feel like it represents a pretty big chunk of the future for web devs. Best of all, I think whether you use Polymer, React, Angular, or anything else, we can all be happy together in the common ecosystem that Web Components give us. So this post isn't telling you what religion to pick; it's telling you there's something awesome happening we can all learn from.
\nYou might not need all the information I'm writing here. So here are links and summaries to the individual segments of my 5-part ES6 Web Components Series.
\nPart 1: A Man Without a Framework (this page) An opinion piece on why and how I decided to give ES6 Web Components a shot with no help from frameworks or libraries.
\nPart 2: The Building Blocks A look at what I mean when I say Web Components and all the pieces that make that up.
\nPart 3: Making an ES6 Component Class How to make a real ES6 Class by extending HTMLElement and making a proper component.
\nPart 4: Project Setup and Opinions Opinions I've formed on project setup as I experimented with rolling my own components. This covers repo setup as well as common methods in my class.
\nPart 5: Conclusion An important look forward if you read and took the other 4 parts of this series to heart and are as excited as I am about them. The short of it: While Web Components is super promising, take it with a grain of salt and do your own due diligence. In other words, I'm very excited about ES6 Web Components, but I haven't been releasing production cross-platform code. Think about it before you do.
\nAnd so we begin:
\nPart 1: A Man Without a Framework
\nThe web is a steaming mess of Javascript frameworks. It can feel impossible to keep up. You can learn something, love it, but then find a bunch of online hate for it. For me, as best as I can recall, my first legitimate love for a JS framework was Dojo. It was several years ago, so my memory is fuzzy, but it had a runtime, modular bits of code I could asynchronously load when I needed it (with AMD), and what seemed like an extensive UI widget set...well, it got me excited.
\nDojo, might have been a little ahead of its time, though. In an age during which folks copied and pasted snippets of jQuery to make things go, Dojo was a hefty setup. It had a learning curve. Unless you were dedicated enough to get around the infuriating bits of that learning curve....wellllll....back to copy/pasting jQuery code.
\nI'm not against folks copying and pasting jQuery. It's so fantastic that people learn new things, and there's nowhere better to start. Throwing a bunch of code into a file, tweaking little bits here and there - that's how I learn a brand-new technique to this day.
\nAt this point in my JS career, though, I want a good place to call home. A comfy cottage that has decent conveniences. Fun to live in, but with enough maintenance already done for me to carve time out for doing more fun things. I don't like doing dishes or handwashing laundry just like I don't care for managing script dependencies, doing DOM manipulation with cases for every browser, or a whole other host of things. I want to work on my app!
\nMy last comfy cottage was Angular. Angular had an easier learning curve than Dojo...but it still had one. What Angular did better was allow me to create modular bits (directives) that could all work together to make some very cool apps. Lots of times, I could focus on my actual application, rather than on what made Angular work. But occasionally, I'd still have to dig into Angular's weird nuances to work something out. $Scope.digest anyone?
\nWhen I get that invested in a framework like Angular, I stretch my feet out toward the toasty fire and do some cool things with it I don't think others are doing. It starts to feel like home, even if it's a little messy. I invite some people over.
\nAt its heart, though, it's still an Angular cottage. If I'd rented it out to someone for a week, they'd have felt a bit awkward and uncomfortable in it. They might have known Javascript pretty well, but Angular's a platform unto itself, so they wouldn't have been sure what was going on at first. It would've been a great little cottage that made them feel at home, too, but that's because I lived in it before them.
\nThen, of course, Angular 2 was announced. Lots of things I knew went out the window. What the hell!? Also, React was happening. Also, Polymer was happening.
\nSo I thought to myself, well, if Angular 2 is no longer recognizable, why not reconsider this whole cottage thing. Leave all the options on the table and check out the new real estate.
\nReact seems pretty awesome; I even played with it some. The whole virtual DOM thing is a cool paradigm. You update this DOM that's off to the side in memory, and it watches the changes, and if there are things that change, the real DOM is updated. Clever. Lots of folks seem to love it and are building some pretty brilliant stuff with and for it.
\nAngular 2 also seems great. I was skeptical, but I got good vibes when my friend, Adrian Pomilio, presented on it at NCDevCon. Despite changing some stuff around, creating a slight learning curve, and making lots of Angular 1 developers angry, its early stage progress looks promising.
\nAnd then there's Polymer. I started learning Google's fancy Web Component framework at a very early stage (at 0.5). I took Polymer pretty seriously. Web Components just made sense. Yes, it's a set of emerging standards, all rolled up into one buzzword. But the modularity of it....well, it gives me real hope for a freaking awesome way to work and even share our work.
\nNow, the problem with Polymer started for me when the 0.5 release iterated to the 0.8 release. And then the 0.8 release iterated to the 1.0 release. I knew to expect breaking changes between these sub 1.0 releases, but it was too much change to call working with Polymer fun. During the slow times, I could get cool stuff done for my application, but then the next wave came and I had to refactor everything and re-learn stuff that was gelling in my mind. What's worse is I didn't know how to keep track of the Polymer component seed generators or the different components that had different version dependencies. They never seemed to work out in a major upgrade; I'd have to tweak bower files to pull in new versions or old versions, and the errors weren't clear on what was happening when something went wrong during this confusing time.
\nEverything became pretty stable with an awesome 1.0 release, but it was too late. I was already jaded enough to evaluate if I really loved Polymer...or perhaps it was simply Web Components that could build the cozy cottage of my dreams. When I thought about it, I liked some of Polymer's syntax, but they weren't really Javascript convention. Polymer also added a whole slew of methods for dealing with variable/method scope - but that was already custom Polymer. And if I thought about it, if I wasn't creating a Material designed application and wanted to customize how I wanted my application to look and act anyway, why import a paper-button that has a whole host of downstream dependencies? Especially when a custom, CSS-styled button tag would suffice?
\nSo I got the brilliant idea to take on Web Components on my own. No help from a framework. I'd use HTML Imports, the Shadow DOM, and more with plain old Javascript. Except I wanted to learn ES6, too, so maybe I'd go a little fancier than plain old Javascript. And oh yeah, I'd need to polyfill for browsers that don't support Web Components, so maybe some help from webcomponents.js (the polyfill that spun off from Polymer).
\nThe further I've built it up, the more I've settled into my current comfy cottage. Sure, I have to plug some holes other frameworks already fixed, but I can look to them for inspiration and guidance. What seems common to all of these frameworks is the convergence on some sort of Web Components. Whether it's true Web Components or not doesn't matter; I can pull ideas from a host of frameworks that all seem to agree on modularity and custom element creation. And of course, an added bonus is learning how everything works under the hood. There were some design decisions, especially in Polymer, that had me scratching my head until I learned how the browser actually dealt with components.
\nWhat's more, with modern browsers, lots of housekeeping problems have gone away. Cross-browser needs that jQuery answered have disappeared if the target browser is recent enough. The application lifecycle has become that of a component, and modularity can provide the foundation of an application structure. If that isn't good enough, I can sprinkle in libraries to help that aren't entire platforms of their own like Angular, React, and Polymer are.
\nEven better, since we're talking Web Components, we can hypothetically share components between React, Angular 2, and Polymer. That's a pretty great place to be.
\nIn my next article, I'll talk coding specifics. But I wanted to explain how I landed here with this one, and impress on you that Web Components aren't just the next passing fad. You might say Angular, React, or Polymer aren't either. I certainly won't argue; they seem legitimately cool. But I find it much safer to rely on web standards and plain Javascript. You might say Web Components aren't a standard yet. Neither is ES6. You might say Chrome is the only browser supporting Web Components as the rest rely on polyfills. Yes, Web Components may sound riskier than the other so-called passing fads, but with the Google-supported polyfill that Polymer uses (webcomponents.js), I feel it's worth the risk right now to learn and play with something that has serious potential down the road. Go ahead, put the tea kettle on over that fire.
\n(And an exciting note: Webkit just announced Shadow DOM support!)
\nIf this pans out, we'll know the underpinnings of how most future web technology works. Polymer works on Web Components. Both React and Angular have hinted at basing future tech on them, but they are smart enough to not bet the farm on it yet.
\nIf it doesn't pan out, well, if you follow along with this series, you'll have custom built this technology with your own Javascript. We can both simply take the bits and pieces that do pan out and adapt.
\nCouldn't it be cool to be a man or woman without a framework?
\n\n", "date_published": "2015-10-27T00:00:00Z" } , { "id": "https://benfarrell.com/blog/2015-06-30-tron-for-githubs-electron/", "url": "https://benfarrell.com/blog/2015-06-30-tron-for-githubs-electron/", "title": "Tron (for Github's Electron)", "content_html": "I'm a big fan of Github's Electron lately. Electron is the underlying tech behind Github's Atom Editor, which they have kindly made open source....YAY!
\nElectron marries Google's open source browser Chromium, with IO.js (the fork of Node.js). What you end up having here is a desktop wrapper that lets you do all sorts of HTML5 goodness with the power of Node.js which lets you access your local system, run C++, and everything else Node/IO.js does.
\nTo get started in Electron, you'd need to do several things:
\nNone of these are that bad, but like any other ecosystem, you need to know how things work and figure out what pieces of the puzzle need to be in place.
\nEven for me, who is pretty good with how to scaffold an Electron app, I prefer not having to do the same things over and over again - so my needs are similar to those of a beginner, in that I just want a quick way to create and run an Electron project.
\nI'm also a big fan of Polymer. The main thing I love about Polymer is how it enforces everything to be a Web Component. You can read all about it's encapsulation model with the Shadow or Shady DOM elsewhere, but I like the fact that Polymer strongly suggests that each component you have is self-runnable. Of course, this isn't exclusive to Polymer, but it's the first time I've really seen self-running components en masse.
\nSo, I've taken some inspiration from Polymer...
\nWhat if you can not only run your main project in Electron, but also run any components you want as standalone Electron applications?
\nWith that, let me tell you about my new CLI tool call "Tron-CLI". Tron is a tool that you'd install globally under node like so:
\n\n\nnpm install -g tron-cli
\n
Once installed, go ahead into a new project directory and type the following into the terminal:
\n\n\ntron create
\n
Tron will download the Electron binaries and create an application folder for you with a fully working dummy application.
\nTo run, you'd typically need to target the Electron executable hidden in the binaries folder and pass in the app folder. I've played around with popping these shell commands into a grunt or gulp file (which is a fine way to go). However, with Tron, you'd simply type:
\n\n\ntron run
\n
The Application Javascript I provide also accepts arguments via the Tron CLI. So if you wanted to pop open the Chromium developer tools when you run your app, you'd do this:
\n\n\ntron run -d
\n
And like I said, I'm a fan of self running components. So if you have those, and especially if you've created them with the Yeoman Polymer Generator where the demo files are placed in your
\n\ntron comp
\n
Of course, if your component structure doesn't adhere to this scheme, go ahead and tweak the tron.json file.
\nYes, your application may quickly grow into something pretty big and you might outgrow Tron. I am to add more features to the application JS as I need them, but for now this is a good quick way to start an app that you can throwaway or mold as you need.
\nTron does a few more good things, but the above is mainly what I'm using it for now in combo with my Polymer projects. To deep dive, go ahead and checkout my readme, but in the meantime...it's super easy to get an Electron project up and running with "tron create" and "tron run"!
\nALSO...
\nI did just recently become aware of Electron-Prebuilt. Great project - it looks like it installs Electron as a global dependency and allows you to use the CLI tool "electron" to run your app. It will assume nothing about your app and lets you author it however you want. My Tron-CLI is more opinionated with how things are setup and scaffolds and app and dev environment for you based on these opinions. Because of these opinions and application code, it does a fair bit more.
\nAlso, Tron lets you have an Electron install per project, whereas Electron-Prebuilt uses a common one. I wouldn't say that either is right or wrong, just a matter of preference.
\nPlease by all means ignore Tron if it's not right for you!
\n", "date_published": "2015-06-30T00:00:00Z" } , { "id": "https://benfarrell.com/blog/2015-05-05-interstitial-vo-creation-with-text-to-speech-and-ffmpeg/", "url": "https://benfarrell.com/blog/2015-05-05-interstitial-vo-creation-with-text-to-speech-and-ffmpeg/", "title": "Interstitial VO Creation with Text to Speech and FFMPEG", "content_html": "Wasn't I playing with virtual reality lately? Yes, but I got distracted. I'll blog on that later, but for now - I've been looking to do cool things with my music discovery library called "the Shark Attack".
\nOne of those cool things is a longstanding problem I've wanted to solve but never got around to it. See, I deliver a weekly radio show to my good friend Vicky Ryder who runs Codebass Radio. I use my Node.js (and soon to be Electron based) SharkAttack to discover all new music from various blogs and feeds and webpages and then package up a playlist. It gets zipped up and passed on as a full 3 hour show she can run in my Wednesday 1pm EST timeslot.
\nIt's fully automatic, which is great. I push a button, and seconds later (after downloading various new music), I get a radio show! The bad: well it's radio...how does anyone know what I'm playing if I only generate the music files? Typically in radio, you'll have a DJ announcing what's been played every few songs. In the spirit of automatically generating a radio show, lets see if we can automatically generate some voice overs!
\nBonus challenge: VO's aren't terribly interesting without a music bed!
\nSo let's tackle TTS (text to speech) first.
\nSo Chrome has a pretty nice text to speech API. Rather, its the "Speech Synthesis API". Unfortunately, this has two problems. The first is that it's not really browser functionality. It takes your requests and passes it through to Google's servers and THEY do all the work. So really, most of the cool stuff isn't really in your browser, Chrome is just a nice pass through.
\nSecondly, even though I plan to end up in desktop wrapper Electron (which uses Chromium), I really want a Node.js solution (or at least a plain vanilla JS solution) because I want ultimate control over my bits. Seriously, I want to cache responses, write them to files, and mix audio. The Speech Synthesis API doesn't do that AFAIK - it just talks through your browser as you command it.
\nLuckily, there are some Node.js libraries like "node-google-text-to-speech" that directly use the Big G's TTS endpoints....like so:
\nhttp://translate.google.com/translate_tts?tl=en&q=<your spoken text here>\nThe "tl=en" param is fun to play around with, because it's the locale of the text. I've played around with French, Australian, and British. What happens is that the text you get back just cops different sounding accents. Honestly though, plain old "en" seems the best for me.
\nThe resulting response body that comes back contains the binary data (in MP3 format) of your spoken text. It sounds slightly robotic, but better than nothing! You can actually try it right in your browser. Seriously. Try this [link](http://translate.google.com/translate_tts?tl=en&q=Ben%20says hi).
\nAn interesting limitation is the character limit of the text you'd like to have spoken. I think it's 100 characters, but I'm using 80 just to be safe. If you play your cards right, you can use this limitation to your advantage.
\nTo get around it - break your text into separate requests. I've gone ahead and split my text string with the Javascript String.prototype.split() method. I split on spaces: .split(' '). What this does is make an array of words. Keep adding word after word onto your request string until you meet your limit! Overflow goes on the next request. And the next request. And the next request.
\nHow can you use this to your advantage? Well, sometimes words can blur together. Google isn't the smartest in terms of speech pacing and word separation. If I'm calling out a song, I don't want to say "andnextupisMySongbyMyArtist"! You really want to pace out that sentence better. There's no way better than to cause Google to lose sentence context by breaking up the sentence text parts in different requests. This results in something more like: "andnextupis.....My Song....by....My Artist".
\nTo do this, I've introduced a loose tag that looks like this:
"And next up is
Of course lots of string parsing ensues! Here's a gist based on the current state of my code. A couple of notes here - I pass on the response to another piece of code that listens and write it to a file. Be sure to write the file with base64 encoding. Second note: the maximum character limitation is because I'm not paying for Google's services. This makes me a little nervous, so, I'm caching speech data on my local filesystem. If a duplicate request is made, I query the file system to see if the data exists. If it does, I don't make the request, but instead load the file data in its place.
\nAnyway, subsequent request data accumulates and accumulates. I concatenate all the data to form the very text that was requested (maximum character limit be damned!)
\nSo next up....I have a voice over. But it's pretty bland. There's no audio backing to it. No music. This isn't very radio show like, is it?
\nSo FFMPEG is awesome. I've said so in previous posts. I've used it to simply convert a video file to an audio file (MP3). But it's SO MUCH MORE!
\nIt took me a while to get there, but here's what I want:
\nAnyway here's the FFMPEG command I figured out:
\nffmpeg -i vo-block-1.mp3 -i VO_musicbed.mp3 -filter_complex "[1:0]afade=t=in:d=5,afade=t=out:st=30:d=5[BED];[0:0]adelay=7000,a pad=pad_len=40000[VO];[VO][BED]amix=inputs=2:duration=shortest" -ar 44100 out.mp3
\nSo there's a few fairly obvious features here. The first are the "-i
The second easiest to explain is the audio bitrate. The -ar flag is set to 44100, or 44.1khz. I had to explicitly set this because Google's Text to Speech bitrate is super low. It doesn't NEED to be high, but guess what? When we mix this and the music bed together, the whole mix cops the super low bitrate! My nice audio bed that I composed myself (thank-you-very-much) sounds like crap! So I tell FFMPEG to force this bitrate. The text to speech gets upconverted, but my music remains mostly the same. Whatever....it sounds good.
\nOk, next we're gonna talk about the whole "-filter_complex" deal. Basically what we have here is a complex filter. We're doing lotsa stuff.
\nFrom the FFMPEG docs, we're talking "filter graphs" and "filter chains". Filter chains are the easiest to wrap your head around. They take an input, and runs various filters on it (in order).
\nSo, for example, I have my fade in/fade out filter chain:\nafade=t=in:d=5,afade=t=out:st=30:d=5
\nThe input (which I'll get to later, when talking filtergraphs), fades in with a duration of 5. The next filter that accepts this faded-in input, then takes the same audio and fades it out. The start time is 30 seconds, and the duration of the fade out is 5 seconds. We know this takes place sequentially because it's a chain - it's separated by commas.
\nLikewise with the speech delay and end padding:\nadelay=7000,a****pad=pad_len=40000
\nHere, we have a filter chain (again separated by commas) where we delay playback of the voice over by 7 seconds and pad the end of the voice over with 40,000 samples. The 40,000 here is fairly arbitrary as I played around with it. We can calculate it based on the samplerate of the audio. For example if it was 44.1khz (or 44,100 hz) these 40,000 samples would be 4,100 samples short of a second. It's kinda unfortunate that the pad length needs to be in samples.....but here we are.
\nThe next filterchain is the mixing of the audio sources:\namix=inputs=2:duration=shortest
\nSo this is easy...we're mixing 2 inputs, and the resulting mix duration is the length of the shortest input.
\nNow what about filter GRAPHS? Well these are semi-colon separated. They are chunks of work to be done with a specified audio input and audio output. They don't really need to be specified in a particular order. Unlike filter chains, with their in-order comma separated commands.
\nA so called "input pad" and "output pad" are written with brackets: [myinputoroutput].
\nIn the case of our fade in/out:\n[1:0]afade=t=in:d=5,afade=t=out:st=30:d=5[BED]
\nHere we're taking input 1:0 (one of the inputs which I found the label for by looking at FFMPEG output in the console), running the filter chain, and outputting that to an "output pad" that I have personally named "BED".
\nAnd then with the delay and padding:\n[0:0]adelay=7000,a****pad=pad_len=40000[VO]
\nHere I'm taking the other input (the VO which is labeled 0:0), running my filter chain on it, and then outputting that to another "output pad" that I've personally named "VO".
\nAnd then the mixing:\n[VO][BED]amix=inputs=2:duration=shortest
\nThis is easy. I'm just taking my two custom named outputs from the other filter chains, and mixing them! Normally, if you didn't specify the "input pads" it would pull from your two input files, but I wanted to process them first.
\nSo that's that. Between the text to speech, and the FFMPEG, I wind up with something like this:
\n[audio mp3="https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2015/05/vo-block-8.mp3"][/audio]
\n", "date_published": "2015-05-05T00:00:00Z" } , { "id": "https://benfarrell.com/blog/2015-04-28-the-weird-developer-ecosystem-for-the-amazon-echo/", "url": "https://benfarrell.com/blog/2015-04-28-the-weird-developer-ecosystem-for-the-amazon-echo/", "title": "The Weird Developer Ecosystem for the Amazon Echo", "content_html": "Update: @garethwatts tells me that we were told that your app doesn't have to be hosted on AWS, only that it must be publicly accessible. In my opinion, this is BETTER, but still not great for just serving off local host in your dev environment! Thanks Gareth!
\nBefore you read this article, I gotta warn you - this isn't a how-to. These are initial impressions of the ecosystem based on a short 45 minute presentation given at an "Amazon Pop Up Loft Event". There's not too much technical detail here - and honestly, it's cause there wasn't too much technical detail at the event.
\nThat said, the Pop Up Loft did paint a rather disappointing picture of making "apps" for the Echo. For those of you who don't know - the Amazon Echo is approximately a one foot tall cylindrical device. It functions like Apple's Siri, Google's voice input, or Microsoft's Cordova. The difference here is that you put it in a room, and it's always on. Simply call out "Alexa, play NPR", and the device will pop on the live streaming NPR station. It will respond to a wide set of questions like "What's the weather", or "How's my commute", and it will talk to you - delivering info in a friendly voice.
\nDifferent folks have different results with it. Personally, the types of questions I want to ask, it can never seem to answer. But, another developer friend of mine has said it's pretty useful for him.
\nI totally think it's fair to expect mixed results from a technology like this that is only released invite only right now. Amazon is rolling it out slowly. For me, however, I get some pretty good use out of it as a capable audio device with decent speakers that I can talk to and play radio stations and podcasts. For $100 (Amazon Prime Members get the $200 price tag slashed in half), I still think it's worth it - and I was excited about the upcoming developer possibilities with the SDK.
\nWell, after tonight's event - I've realized that the SDK isn't REALLY an SDK. I mean I guess in the strictest sense, it might be, but the Echo is not really a hacker's friend. It doesn't appear to be an open device that you can tinker and play around with. Instead, you are required to make "apps" for it.
\nSo what does this mean? Well, the event tonight portrayed some architectural diagrams and showed some configuration settings through Amazon. Here's the low-down:
\n"Apps" are simply web services built on AWS. When you call upon Alexa to "use" a specific app (such as "Alexa, do
Amazon has an "intents" JSON configuration that you can edit via their admin console. You set up certain phrases, and swap in keywords to those phrases my marking them with curly braces and pipes. For example, you'd have a long list of phrases in the settings:
\n"What is my horoscope for {virgo | horoscope}"
\n"What's my horoscope for {virgo | horoscope}"
\nTo be honest, these examples are to the best of my recollection, and I might have the syntax slightly wrong, especially around what it pipes to. Either way, elsewhere in the config, you'd list all the astrological signs, and listing "virgo" in the syntax here in the phrasing is a little auto-magical in that when you put "virgo" in context here, it just assumes you mean all the possible options you defined elsewhere. This rubbed me kind wrong, not very explicit at all - which I'd bet means not very flexible when it comes down to it.
\nAnyway, the implementation is a part of this, but given that it'll probably evolve AND I could be remembering things slightly wrong, I won't harp on it. What I will harp on, is the entire app model they are forcing here!
\nUpdate: I'm probably wrong on needing to host your app on AWS - I have a note at the top of this post about this
\nSo, first of all - you gotta use AWS to deploy your app. So look, I'm not all that familiar with AWS. I mean, I know what it is, but I haven't used it to deploy an app. I've used other services, some in the cloud, some not - just not AWS. Here, you're forced into deploying on their cloud. For me, this means signing up and presumably paying for Amazon's ecosystem.
\nAs good as AWS is, I'm not allowed to develop something locally on my own network. I'm definitely not opposed to delivering a production application on a solid infrastructure like AWS, but if I'm just playing around, I don't want to go to that hassle! Also if I'm an enterprise customer with my own locked down network, I can't deploy my app inside my firewall.
\nOnce you get your app up and running you'll need to create an official Amazon marketplace app. This is the same channel you'd use for developing a Kindle App, a Fire App, anything they distribute on their marketplace. It needs to use https, so you'll need a certificate, though they do allow self signed ones for non-production/developer apps.
\nAs a developer, supposedly whatever you create will be available on your Echo by default. Distribution to the wider populace is murkier. Amazon couldn't comment on how apps would be distributed in the future as its subject to change, and they didn't seem quite sure if you'd enable it on your phone or what. The speaker did mention that it was too difficult to allow "install" via voice commands.
\nOK so - you see my problem yet? Maybe I'm a front end developer or even a code proficient designer. I have an Echo and I want to design an experience around it. It seems simple enough. How hard could crafting some code around questions and responses be? Well not very - except I need to learn how to do freaking cloud deployment now! And guess what, just to play with this thing, I'll need to enter my credit card info, register for the service, and spin up an instance.
\nOne thing that was a little disconcerting as well, was that the presenter made a point of showing us that the lag time if you only spin up ONE instance of your app might make the Echo time-out on a response, so he emphasized the need for 3 or more for reliable usage.
\nLike I said, I've not used AWS specifically, but I'm guessing this adds up to real money. Perhaps it's just pennies, but I've looked at their pricing structure before and been a wee bit confused on how much I'd be charged for doing anything. Again, I have no question that AWS provides a great service for your production infrastructure, and I'm quite sure it's worth the price. But as a developer who's just playing around, I don't know what I'm getting into.
\nThe other aspect I have question with is the whole notion that these are "apps". Yes, I get the app ecosystem. It works for Apple. It trickled down from Apple and it works for a lot of different ecosystems now.
\nHowever, when you think about so called "micro-transactions" with these limited devices, how do you monetize? Why would a developer be inclined to actually make an app? I first heard the concept of a "micro-transaction" in reference to Google Glass. It's a quick 2 second look to catch information, and then you're done. This sounds a lot like the Android and Apple Watches that are coming into play.
\nI'd wager that most of the monetization of Watch apps aren't done for the watch at all. The Watch is an extension of the main app you'd purchase that lives on your phone. It's an extra compelling feature on an already good app. If you don't purchase the app, you might have some ads on your phone. I can't imagine users would take kindly to ads embedded in microtransactions - because it's a tiny stream of only the info you need for a limited few seconds. Extra info that you don't need like ads would just be too much.
\nI'd describe your interactions with the Echo in the same way. They are microtransactions. There isn't extra info, extra ad space, anything. You ask for what you need and you get it. The Echo DOES have a companion app, but aside from settings and setup, you don't really end up using it. So with microtransactions being the ONLY window into your app, how do you monetize? How do you embed ads? Will people pay purchase simple voice apps? And, given how unsure Amazon was on distribution, will Amazon even SELL simply voice apps?
\nIt seems to me that the only way to make money here is to help existing brands extend their presence onto the Echo. The notion of developing a killer app here seems quite limited unless you do it for free. And even then, you're paying to use Amazon's infrastructure to do it!
\nIn the end, I'd much prefer that the Echo uses the Amazon cloud to simply convert speech to text and vice versa. If the device itself provided an SDK that you could play around with on your local network to work with the plain text responses, it would be AMAZING!
\nI'd rather have a cool robot buddy in my living room, than an app store. Oh well. Either way, I'll probably give it a try once my developer invite comes in. But I'm not as psyched as I was.
\n", "date_published": "2015-04-28T00:00:00Z" } , { "id": "https://benfarrell.com/blog/2015-04-05-who-is-managing-your-project/", "url": "https://benfarrell.com/blog/2015-04-05-who-is-managing-your-project/", "title": "Who is managing your project?", "content_html": "I've had some pretty awesome co-workers in my time, so no doubt I keep in touch (or at least stalk them a little on Facebook and be glad when they are doing well). One of those folks is a very awesome lady named Polly Searles. To be honest, I don't recall working with her THAT much at my past job, but I do remember her to be crazy good at managing whatever project I was on.
\nThe flashbacks are coming because she was recently profiled in her current company's bio section. Fablevision is certainly lucky to have her (as well as several other really awesome past co-workers). I gave her bio a quick read, and while it does great to reflect her achievements and personality, it did something else for me - it caused me to reflect on how weird it was that for 8 years of my professional career I was not working with Project Managers, Agile Leaders, Scrum Masters, or any of that....
\nI was working with Producers.
\nIn 2008, when I left what I described as "Agency" work to a startup, and I was just getting agile methodology tossed at me, I really didn't think it was strange to describe that I worked with "producers". I thought it was just another type of project manager that people would be familiar with. Now that I'm reflecting, I'm realizing how strange this might be to some folks. In fact, I'm reconsidering categorizing my past jobs as "Agencies", but rather they were Television Production jobs, despite the fact that I was a developer/designer.
\nFrom 2000-2008, my career was spent working on software and games and websites. The common thread behind all of this, were that we were working with TV properties. Reading Rainbow, Mr Rogers, Sesame Street, Nickelodeon, etc. For 8 years, I was working with "producers". As internet experiences became more and more capable, TV was spilling over and asking for more and more companion websites, animations, and games.
\nRather than break from tradition, it would seem, TV producers would now be asked to help manage the new medium. It seems that video games follow a similar path. Generally speaking, it seems like if there's animated, scripted content - you're working with a producer.
\nWhat did a producer do? Well, from my experience as a developer, they were project managers. At least that's how they appeared at the time. I've since gone on to work with very capable product managers, and normal PM's just didn't take on as much as a Producer does.
\nThe Producers I've worked with managed projects, but also were the face of the project when interacting with clients. They responded to RFPs (request for proposals), even planning and scoping mini prototypes ahead of the work to impress potential clients. The producer made creative decisions for projects with input from art directors, designers, and developers. They single handedly brought people together around a project, doing whatever grunt work it took to pull things together. My best times working with producers were when they utilized the entire team for input and we made decisions together. It led to some awesome work.
\nAfter leaving TV behind, I joined a startup. We did TV! Its funny, but even though it was still television, we were approaching from a very technical angle - which was around video search, publishing, and metadata. No longer did I work with Producers. We weren't creating scripted content, just working the playback and discovery angle. For project organizations, I worked with a few Project Managers (PMs).
\nI worked with some great PMs, but I didn't really enjoy the relationship. It was a very segmented one.
\nYOU DEVELOPER: YOU CRANK OUT CODE
\nI PM WILL TELL YOU WHEN THINGS ARE DUE, AND YOU DO THEM.
\nIt's not a relationship that fosters creativity. I certainly liked having someone on point to figure out budgets and work with the client all day so I didn't have to. However, when you the developer hears about client requests second hand, there's no room for you to have much input or creativity. Instead, you're just doing the job - kinda like a robot. It's hard to go above and beyond.
\nAs the company got a little leaner and our products got more formulated with some great product managers - I got left outside of the product team and on the professional services team. Here, the best thing ever happened to me - I started working with a PM (also the VP of Engineering) and a couple Account Service folks that were way too busy. They knew they were too busy, so I'm not badmouthing them - in fact, if you'd ask me who were my favorite people helping me to manage my products, I'd instantly point to them and a couple of Producers I'd worked with.
\nAnyway, because they were too busy, I was put into the position to interact with our clients a lot. I got to hear the project needs, putting me in a great position to make suggestions, get the clients to back down on stuff that didn't help them or make sense, and overall put my knowledge to good use rather than being isolated where a PM tells you: DEVELOPER, DO THIS WORK NOW BY MONTH'S END.
\nThe overworked PM/VP of Engineering ended up adopting some Agile practices. Everyone hates on agile these days, but at this job it was a great deal. We'd all help write user stories and tasks. Everyone would agree on them and help to elaborate them better when we could. QA was involved from the beginning. Design was involved from the beginning. It was a freaking awesome team effort. Sprints were 2 weeks, we all realized it was somewhat arbitrary, but we didn't care. Real deadlines were met as we occasionally mocked our 2 week sprints when they didn't make sense.
\nWe made Agile work for us in our unique projects and deadlines.
\nAfter this job, I got a rather large corporate job. I'm still there, so I shouldn't cover any of the missteps which have since been corrected - but suffice to say project management should not be one size fits all. It should also be more of a team effort with a designated leader to make decisions and move things along.
\nOne very recent project involved me working with a Design Researcher on a very quick UX prototype. She was/is awesome - but it was very reminiscent of my best times working with a producer. She took on the roles of researching what we should actually build, interacting with our clients, and being very inclusive of our small team every step of the way. There was tons of freedom here to be creative and listen to the unique needs of the project. You can probably guess it was a huge success.
\nAnyway, point here is that my time with Polly and other great Producers have really made me think about what a successful mechanism is for project organization. I'd emphasize the following:
\nI'm sure I'll have my mind blown in the future with some other awesome way of managing a project. But I'm also sure I'll just say: "Hey this reminds me of the awesome times I had working with a Producer!"
\n", "date_published": "2015-04-05T00:00:00Z" } , { "id": "https://benfarrell.com/blog/2015-03-24-codebass-radio/", "url": "https://benfarrell.com/blog/2015-03-24-codebass-radio/", "title": "Codebass Radio", "content_html": "Codebass Radio is "Where Geek and Music Combine". It's an awesome developer/geek/nerd oriented internet radio station founded by Vicky Ryder. I'm an eternally grateful to be her Senior Producer at the station - trying to step in and help where I can, as well as run two radio shows of my own.
\nCheck out CodebassRadio.net or listen live right now - There exists a show for every taste, and the schedule is chock full or fresh content every workday.
\n![\"codebassheaderlitenew\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2011/12/codebassheaderlitenew.png\")
This past year and a half have been a little busy for me. I've moved to the Bay Area, moved again to Oakland, started a new job, and had a steady stream of speaking gigs at conferences.
\nIt's now March 2015 and my blogging has dropped off quite a bit. Back when I was doing this regularly, I really enjoyed learning something new and sharing it right away so someone else could learn as well. Hell, I've even gone back to my own blog posts to remember how to do something.
\nPrior to this, my blog had been getting heavy on the JS frameworks. I got quite a bit of traffic for the various Angular posts (in fact, folks are still visiting them quite a bit today).
\nAnyway, if you've been stopping by for the JS framework posts, THANKS! Frankly, though, as I've explored several application frameworks, there's a bit of a revolving door as the new hotness is released. I'm a bit bored of exploring frameworks just because they exist. Personally, I just want to create - and the framework used is getting less and less interesting to me.
\nI may blog a little bit more about Polymer and Web Components, my new favorite creation tools. But what I'm REALLY interested in these days is 3D and VR. I'm far from an expert yet, but I'd love to share my experiments and thoughts as I continue.
\nAs my schedule has died down a bit, I have some projects that I plan to pick up again, and a few more ideas I plan to start. Continuing forward, I definitely plan to keep working on SharkAttack and keep dropping weekly shows on CodebassRadio. I've got some cool ideas for this, my music aggregation tool. I'm also psyched to continue on with Sweatin to the Web. Being in the Bay Area, with lots of health and fitness startups could be awesome for this if I play my cards right. The first season of SttW was short and way too long ago, but I expect to start production on Season 2 this April.
\nI've got some new stuff in mind, too. I've been talking about writing a book forever, and its time to stop talking and start doing. I opened up a Gitbook account back in November, and perhaps I'll blog a bit about trying to write something using Git. I've got a few other ideas for little projects here and there - but as I've mentioned, the new hotness is VR. I plan to incorporate some experiments with Sweatin to the Web and perhaps see where fitness and VR can intersect. VR certainly fits with getting up and moving (though maybe not much).
\nEither way, I'm excited to usher in 2015....albeit in March. But you know, I was busy! I just got cracking this weekend with a new minimalist blog look with actual GOOD typography. That little bit right there is enough to get me excited about the year ahead and motivate me to get going on all the stuff I wanna do.
\n", "date_published": "2015-03-24T00:00:00Z" } , { "id": "https://benfarrell.com/blog/2014-10-21-atomic-desktop-power/", "url": "https://benfarrell.com/blog/2014-10-21-atomic-desktop-power/", "title": "Atomic Desktop Power", "content_html": "Hey everyone! I just got back from HTML5DevConf in San Francisco where I gave my talk "Atomic Desktop Power with Node.js". I had also given the same talk on the east coast last month at NCDevCon.
\nAs promised, I want to share some links with you. Like I said in the talk, I made my own slide show display all in Atom-Shell itself using Polymer. Here's the source: https://github.com/bengfarrell/atomicdesktoppower.
\nThough, the slide deck ITSELF was an Atom-Shell demo, I did have some more targeted C++/Node.js addons demoed. The first was a simple "hello world" addon, followed by a library that generated random 2d/3d coordinates to make a red div box move in my HTML page. The third was a performance test showing how fast we can count to a billion with Node.js vs a Node.js C++ addon. All of these demos can be installed using apm/npm inside my slide deck, but they are also part of a separate standalone repo here: https://github.com/bengfarrell/genericnodeaddon.
\nI also used/mentioned NAN, or Native Abstractions for Node.js" to deal with the changeover happening from Node v0.10 to v0.11 when you write C++ addons. And of course, since Atom-Shell uses v0.11, it was important to my talk!
\nI also briefly mentioned OpenCV for Node. I showed a quick facial recognition demo working from my Atom-Shell based slide deck. You can install this over NPM or grab it from Peter Braden's github repo: https://github.com/peterbraden/node-opencv. Also, while OpenCV is cool, it's a bit more difficult to install on Windows, so I documented it all here: https://github.com/bengfarrell/installing-node-opencv.
\nLastly, if you're still confused what Atom-Shell is (I hope you're not!), you can check out the Atom-Shell page and documentation here: https://github.com/atom/atom-shell/tree/master/docs. Downloadable releases can be found on this page: https://github.com/atom/atom-shell/releases. Differences to Node-Webkit came up, and you can see the documentation that Github wrote talking about that: https://github.com/atom/atom-shell/blob/master/docs/development/atom-shell-vs-node-webkit.md.
\nThat should do it! If you came to my talk, thanks so much for coming. It was tons o fun, and I was quite happy to meet and talk with everyone that came up to ask questions afterward.
\n", "date_published": "2014-10-21T00:00:00Z" } , { "id": "https://benfarrell.com/blog/2014-07-28-working-with-input-text-bound-to-a-model-in-polymer-made-me-angry/", "url": "https://benfarrell.com/blog/2014-07-28-working-with-input-text-bound-to-a-model-in-polymer-made-me-angry/", "title": "Working with Input Text Bound to a Model in Polymer....made me angry", "content_html": "Gah! Ben smash. I wanted to do the simplest thing in Polymer, only to find it was RAGE INDUCING! But now, at least I have a good understanding of how filters work - yay!
\nOK so here's my problem. Given a JSON model...
\n\n { "id": "somekindofstring",\n "maxAge": 289,\n "maxItems": 21,\n "maxDuration": 600,\n }\n... I'd like to create a few text input fields that allow me to edit my values and two-way bind to my JSON model. For those that aren't up on binding terminology, two way binding means that when my input field appears on the screen, it picks up the value in my JSON data. Then, if I edit the field on my page, it automagically updates the data model.
\nMy "id" field is super easy because it's a string. In Polymer, it's as simple as this:
\n\n<paper-input label="id" floatingLabel value=""></paper-input>\nExcellent right? "Paper-input" is just an input field that has been snazzified as a component by Polymer to be made better. By putting what I'm binding to in double curly braces, the input field will immediately show "somekindofstring" as the value of the input field. What I haven't shown you is that I've already loaded my JSON data, and put this object in my Polymer scope so that it's easy to bind to. Something like this:
\n\n Polymer('mycomponent', {\n mymodel: { "id": "somekindofstring",\n "maxAge": 289,\n "maxItems": 21,\n "maxDuration": 600 }\n ....\n\nI dumbed it down a lot. I actually load the JSON data from a separate file....and it's an array, but for the purposes of showing binding, I don't want to get too complicated.
\nAnyway it works! Let's try the same on one of my numeric values with a snazzy Polymer slider:
\n\n<paper-slider min="0" max="365" value="" editable></paper-slider>\nThis also works! I have a slider that goes from 0 to 365, and binds to the "maxAge" property of my data. Perfect, right? Well not quite...check out my new data model after it's been updated:
\n\n { "id": "somekindofstring",\n "maxAge": "289",\n "maxItems": 21,\n "maxDuration": 600,\n }\nOh no! My "maxAge" value is now a string! To figure things out, I turned to some Polymer documentation on "Expressions". The first thing I tried was a unary operator. From the docs, it looks like you can use a + sign to convert something to a number. So I gave it a shot:
\n\n<paper-slider min="0" max="365" value="NaN" editable></paper-slider>\nNow, that sorta worked....I think. Seems to bind OK - but my slider goes a little crazy and my two-way binding breaks. I'm not so sure that this plus sign works well for two-way binding. So, let's try the next thing...a custom filter!
\n\n<paper-slider min="0" max="365" value="{{mymodel.maxAge | toInteger}}" editable></paper-slider>\n\nThis is the part of Polymer that lets you define your own filter. To be clear..."toInteger" doesn't actually exist. I need to create it myself. From Polymer's documentation, I can do this:
\n\ntoUpperCase: function(value) {\n return value.toUpperCase();\n },\nEasy enough, right? I can adapt this filter (which is placed right on my current component's scope) to an integer filter like this:
\n\ntoInteger: function(value) {\n return parseInt(value);\n },\nGood? Unfortunately, no. Something ends up broken. I get the message "Cannot find function or filter: toInteger" when I move the slider, even though it's OK on the initial binding. So there are a few things going wrong here. The "toUpperCase" example in the Polymer docs isn't quite complete. It actually breaks when you try it!
\nHere's my implementation of their example:
\n\n<div>{{mymodel.id | toUpperCase }}></div>\n\ntoUpperCase: function(value) {\n return value.toUpperCase();\n},\n\nThe error I get says this: "Uncaught TypeError: Cannot call method 'toUpperCase' of undefined". Thanks to a helpful StackOverflow answer I know now, that Polymer doesn't handle things well when the value is undefined or null. I'm guessing that when the initial component is loaded up, it doesn't have the data yet. Of course, since I'm loading my data from an external source it probably doesn't exist when the my binding requests it. So, we need to handle it:
\n\n toUpperCase: function(value) {\n if (value) {\n return value.toString()\n } else { return "" }\n },\nCool? Yep...for the string based input field - but there is yet another aspect to string/integer conversion. Polymer offers a little more advanced functionality to define exactly how things render to the DOM and how they render to the model. So let's use this to tell Polymer to render the number from the data model as a string when going to the DOM, but render the string from the DOM as a number when going out to the data model:
\n\n<paper-slider min="0" max="365" value="{{mymodel.maxAge | toInteger}}" editable></paper-slider>\n\n toInteger: {\n toDOM: function(value) {\n if (value) {\n return value.toString();\n } else { return ""; }\n },\n toModel: function(value) {\n if (value) {\n return parseInt(value);\n } else { return 0 }\n }\n },\n\nAnd so there it is! Hopefully there are simpler ways to do this, but I couldn't find any. When you know how Polymer works - this DOES make sense, and it reveals a very expressive way to do custom filters. Secretly I'm hoping that the two-way binding using NaN is a bug and will be fixed someday. In the meantime, I have something that works!
\n", "date_published": "2014-07-28T00:00:00Z" } , { "id": "https://benfarrell.com/blog/2014-07-22-managing-your-polymer-project-with-git-bower-and-yeoman/", "url": "https://benfarrell.com/blog/2014-07-22-managing-your-polymer-project-with-git-bower-and-yeoman/", "title": "Managing your Polymer Project with Git, Bower, and Yeoman", "content_html": "I've been working more and more with Polymer lately and trying to establish some workflows that make sense. For those that aren't familiar with Polymer, it's essentially a set of polyfills that enable you to work with those fancy things called Web Components even though browsers don't generally support them yet.
\nAll the Javascript frameworks/tools/libraries I've used to date are fairly simple to manage. Your Javascript is over in one place, your CSS in another, and your HTML/templates in another place. This kind of setup enables you to structure each piece separately and exactly how you want. For example, in an Angular project, I like to have a "scripts" folder. In that folder we could have more subfolders: "controllers", "directives", "services", and more. If your project is particularly complex, maybe you'll have subfolders to organize your directives better. Regardless of how you organize things, you'll probably just concat/minify in some sort of build step - or play it loose and just have tons of script blocks in your HTML file.
\n![\"Screenshot](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2014/07/Screenshot-from-2014-07-21-220258-e1406005541236.png\")
Above you can see a simple AngularJS application structure, with perfectly organized Javascript. I'd say my CSS styles are equally organized, but this project is fairly tiny, so I just have one medium sized CSS file in that styles folder.
\nThis may be what you're used to, but think about the following:
\nLet's say I'd like to edit things relating to my game's scorecard. You can see the controller in scripts/controllers/scorecard.js. But what about the template it uses? Or the style? How would you locate related media? Well, it's not THAT difficult, but would involve you digging through some code to find out what CSS styles it uses, what HTML templates it uses, and what directives it uses. You could be reaching into a lot of different organizational structures to track down how to make edits to that single scorecard piece.
\nIf you know Angular well enough you and your team might agree on some best practices, and what I outlined above isn't that difficult. But what if you've never seen Angular before? What if your team doesn't agree on a structure and goes off and does other things?
\nAny workflow can go badly - but consider Polymer/Web Components for a bit. Each "component" is a standalone piece that contains it's own HTML, it's own CSS, and it's own Javascript. Breaking each piece of your application down into a simple component means that you can easily see and group a chunk of functionality by a component folder. No more hunting for what CSS controls your style on your HTML view - it's right there in the same folder. A component can be as simple as a button, and get as complicated as a entire application. Both examples are simply components, though your application component might import and depend on your button component and lots of other things.
\nLets take a good look at what a component is comprised of. This is a good time to bring in Yeoman and let that be your guide.
\n\nNormally, I'm not much of a Yeoman user. On Angular projects it tends to give me lots of stuff I don't need and have to throw it out. It's like getting a Dell computer and having to get rid of all the awful garbage they pre-installed. I kinda feel the same way about Yeoman. When I first start with a tech like Angular or ThreeJS, Yeoman can give me a nice starter application to help the learning curve. However, once I feel competent, I tend to throw out Yeoman completely. Of course, Yeoman saves you some time when starting your project. Often times, it can be a little difficult to remember all the nuances you need to start something. But, me - I'll just rely on past projects and copy relevant bits over. If I started new projects ALL THE TIME, then yes, I might make my own little Yeoman generator.
\nHere's the thing with Polymer: You are starting tiny little projects ALL THE TIME. Every new component you make is like a tiny project! Thankfully, the Polymer Yeoman generator isn't like a Dell loaded with garbage. It produces a nice slim project full of only the bits I care about! Let's take a look at those bits to get a feel for what a component is:
\n![\"Screenshot](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2014/07/Screenshot-from-2014-07-21-223356-e1406007319451.png\")
So, here's a component I made. But I didn't create a single file here! Just filled in code where I needed (mostly in my HTML file). First I created the folder, named it "sa-libraryview" (I'll discuss my naming in a bit), and then went into the folder and ran "yo polymer" from the command line. The generator saw the name of my folder and made my component have the same name. It also wired up all my HTML and CSS as a blank (but functional) component. The sa-libraryview CSS and HTML are obviously the functional pieces of my component. It's where I put the logic and styles.
\nThe demo.html file is pretty sweet. Components are used by popping the component as a tag on a page. So, testing your component by loading up sa-libraryview in a browser probably won't work. The demo.html page, though, is auto-generated and includes your component tag on the page. So, right away...without doing anything...you get a free demo page! YAY. You might argue that a bower.json file, a gitignore, jshintrc, and others aren't STRICTLY necessary - but they are tiny and do more good than harm especially if you are doing a good job managing each component as a separate project. Please please please use your README file as well! It's great for documentation.
\nThe only thing I couldn't figure out why it exists, is the index.html file. I was gonna cop-out and just say it's unnecessary. Each and every index.html file just includes one component...something called "core-component-page". I finally just decided to load up the file in my browser. It honestly never occurred to me to bother trying because I expected a blank page. BUT GUESS WHAT?!?!
\n![\"Screenshot](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2014/07/Screenshot-from-2014-07-21-2254002-e1406008608336.png\")
A whole entire documentation page is generated on the fly from my component! It comes complete with a demo button that links to my demo page. If I had done a better job with code comments, I'd have all my attributes, properties, and methods inline here...alllll looking awesome.
\nCool, so that's a component generated from Yeoman. Specifically, it uses the Polymer component seed that the Polymer team created.
\nSo, where do all these components live?
\n\nSo, here's the thing... I recall reading somewhere on the Polymer website that the Google engineers who work on Polymer don't actually use Bower all that often. I'm not certain what their workflow is like, but it could be symlinked/aliased component folders. That's certainly one way to do things, and probably one very friendly to working on a big component set that spans multiple projects. Me though? I'm just working on a single project that pulls from existing Polymer "Core" and "Paper" components that can be pulled down with Bower. I'm also creating my own components that are very specific to my project and I probably won't reuse.
\nBower is weird though, right? It's a very flat structure. All of your bower dependencies are placed into a folder and not nested. Once you start working with a few Polymer components, the dependencies start to pile up. Your component folder becomes pretty lengthy. It would be nice to organize all my "paper" components in the paper folder, all my "core" components in a "core" folder. But, no - that's not how Bower works. In fact, it doesn't seem to be how Polymer wants to be! In each of those Polymer components, if there are dependencies, it will link to it's dependency one level up in a relative manner.
\nHere's an example:
\nPaper-Button depends on all of these things (they are links from the paper-button.html):
\n\n<link href="../polymer/polymer.html" rel="import">\n<link href="../core-icon/core-icon.html" rel="import">\n<link href="../paper-focusable/paper-focusable.html" rel="import">\n<link href="../paper-ripple/paper-ripple.html" rel="import">\n<link href="../paper-shadow/paper-shadow.html" rel="import">\nSo, unless you manually go in and alter the source - Polymer is always looking for it's dependencies in the same relative spot. If you use Bower, everything that gets pulled in will all go in the same folder - the components folder you list in your .bowerrc file. That's fine, I guess. Why fight it? Except for the fact that that I'm counting 41 Polymer components in my folder now! These are mostly UI elements that are handy to have in my project. To make things more manageable, most are prefixed with "core-" and "paper-".
\nThe prefix makes working with your own components a little better. Given that I'm not going to register each of my project specific components with Bower, AND they need to live side by side with the Polymer components because of the linkage, I need a way to manage both my Bower managed components and my Github managed components in the same folder! This is where the prefix and our .gitignore comes in handy.
\n\nProbably the best thing to do is pick a project prefix. My project is called SharkAttack, so the prefix is "sa-". Every single component I make is called "sa-this" or "sa-that". They all live side by side with the "core-" and the "paper-" components that Bower pulls in for me. I'd like to leave all my Bower components alone, and be able to commit any "sa-" components to Github. So....let's use our handy .gitignore file and ABUSE the wildcard.
\nMy .gitignore file:
\n\ncomponents/polymer\ncomponents/font-roboto\ncomponents/core-*\ncomponents/paper-*\ncomponents/platform\n\n.idea\nbinaries\nnode_modules\nSince most Polymer component are prefixed, I can simply use the wildcard and Gitignore all components that start with "core" and all that start with "paper". I have 3 other miscellaneous ones, but it's not bad to have a few extra lines. Of course I ignore my WebStorm IDE files, my node_modules, and my Atom-Shell binary files (but that's specific to my project).
\nYep, so here we have a solution to our bower components and out custom components living side by side. Woot! This is just one possible workflow. If you have a better one, go ahead and leave a comment!
\n", "date_published": "2014-07-22T00:00:00Z" } , { "id": "https://benfarrell.com/blog/2014-07-06-a-polymer-gulp-atom-shell-workflow/", "url": "https://benfarrell.com/blog/2014-07-06-a-polymer-gulp-atom-shell-workflow/", "title": "A Polymer + Gulp + Atom-Shell Workflow", "content_html": "Polymer has been on my mind for a while. I've been quite happy plugging away with AngularJS, however a co-worker of mine mentioned to me that AngularJS just wasn't present at Google I/O with some uncomfortable looks from Polymer presenters when asked about Angular. Given the recent social media blitz of Polymer and Material from Google with a strong set of components and documentation at http://www.polymer-project.org, I thought it was time to give it a strong and serious look.
\nAt this point, I couldn't even call myself an early adopter - more like a late-early adopter. I think we'll see a strong showing from AngularJS for at least another couple of years, but certainly Google seems to be putting its money on Polymer right now to build the platform at a pretty fast clip. As web developers, we should probably expect to be happy with Angular for another good year or so until the next-gen Polymer vs Angular battles force us to make a hard choice at where to take our skills next.
\nCertainly, you can't bet on Polymer taking over - Google has a number of projects that just failed to take off. However, Polymer isn't just about useless buzzwords, it actually seems to have some nice structure behind it. So, I thought it would be nice to give it a shot!
\n\nPolymer's biggest win for me is the idea of loosely connected components (web components). Web Components don't have to be tied to Polymer, but for the purposes of this post (and my lack of exploring outside of Polymer) I'll call them one and the same.
\nComponents can take the form of anything from a simple button, to a scrolling area, to a layout, to an entire application. In fact, for ease of use in the Polymer platform - you really should make everything into a component, no matter how large (your larger components will likely have lots of component dependencies). It enables some nice HTML hooks (like creating your own custom click handlers on elements). In addition to using some nice Polymer features, having everything be a component means that you can test things individually very nicely.
\nIn fact, if you use Yeoman, you can scaffold out new Polymer components very nicely. Using yo polymer inside a new component directory will use the Polymer component seed to make your blank component as well as a demo harness to test out your new awesome component! Normally I never use Yeoman, but in the context of Polymer, I now use it all the time.
\nSo yah, web components are pretty awesome. Testing individual components? Also awesome. When you are able to test each component individually instead of having to click through your end application to test the thing you want to test, it speeds development up a lot.
\n\nGulp is another tool I'm definitely not an early adopter on, but I figured I'd give it a shot! I'm not unhappy with Grunt as a task runner, but everyone seems to think highly of Gulp. So I figured I'd give it a shot while I learn a few other techs. Yet another co-worker says the speed of Less compilation with Gulp is enough to make him a convert , so I say "YES, I WANT TO TRY IT PLEASE!". For the purposes of this post, however, you could easily do the exact same things in Grunt, just with a slightly different syntax.
\n\nOk, as much as I'm not an early adopter of those other two, I jumped on Atom-Shell as quickly as I heard it was open-sourced. Atom-Shell is a desktop wrapper for your HTML/JS/CSS based applications, offering a seamless way to inject Node.js code right into your application. Technically it's Chromium Embedded Framework with a bridge to Node.js. This means, as a very base-level example, I can do direct file-manipulation using Node's "fs" module. It's as easy as requiring the package in your front-end code and using it, just the same way as you would in your NodeJS application. You can mix browser based code with NodeJS code, and obviously it doesn't stop at file manipulation. In theory, most everything you can do in Node can now be done in Atom-Shell. In this way, it's similar to Node-Webkit.
\n\nDoes Polymer work in Atom-Shell? Yes. Seems to work quite well so far. However, Atom-Shell has a main entry point: main.js. Inside this Javascript entry point, you will likely tell it to launch an HTML page for your application:
\n\napp.on('ready', function() {\n// Create the browser window.\nmainWindow = new BrowserWindow({width: 800, height: 800});\nmainWindow.loadUrl('file://' + __dirname + '/index.html' );\nMeanwhile, Gulp can be set up to launch Atom-shell and hit this entry point to launch your application. First, you can setup a Gulp task to download Atom-shell for you:
\n\nvar downloadatomshell = require('gulp-download-atom-shell');\n\ngulp.task('downloadatomshell', function(cb){\n downloadatomshell({\n version: '0.13.3',\n outputDir: 'binaries'\n }, cb);\n});\nOf course this requires you to install gulp-download-atom-shell via NPM first. But once that's set up, you can run the task with gulp, and an instance of Atom-Shell will appear in your project under a "binaries" folder.
\nFrom there, we can set up another Gulp task to launch an instance of your Atom-Shell application. For this, we'll just use the Gulp "shell" command, which you need to use NPM to install "gulp-shell". This command will vary from OS to OS. What I'm showing is something that works on Linux and will most likely work on OSX. The shell command will vary on Windows, where you'll need to reverse the slash and point to an atom.exe instead of just atom.
\n\nvar gulp = require('gulp');\nvar shell = require('gulp-shell');\nvar downloadatomshell = require('gulp-download-atom-shell');\n\ngulp.task('downloadatomshell', function(cb){\n downloadatomshell({\n version: '0.13.3',\n outputDir: 'binaries'\n }, cb);\n});\n\ngulp.task('demo', shell.task([\n 'binaries/atom default_app'\n]));\n\nSo here, we can launch our atom shell application using "gulp demo". This assumes your Atom-Shell app resides in a folder called default_app, but you can change this to anything.
\nNow, the problem here is that we are using a single entry point! Remember how awesome I said Polymer was for pretty much forcing you to have loosely connected components all with their own demo pages? It would be a shame to lose this! Originally, I thought that maybe I should just leave the components to be tested in browser while the entire app itself would be tested as an Atom-Shell app. This doesn't work out to well as we scatter NodeJS code that doesn't work in a browser throughout our components. So, I wanted to figure out a way to test each component inside Atom-Shell.
\nTo accomplish this, I start with Gulp, and pass an additional argument to Atom-Shell:
\n\nvar gulp = require('gulp');\nvar shell = require('gulp-shell');\nvar downloadatomshell = require('gulp-download-atom-shell');\n\ngulp.task('downloadatomshell', function(cb){\n downloadatomshell({\n version: '0.13.3',\n outputDir: 'binaries'\n }, cb);\n});\n\ngulp.task('demo', shell.task([\n 'binaries/atom default_app'\n]));\n\ngulp.task('democ', shell.task([\n 'binaries/atom default_app components/' + gulp.env.comp + '/demo.html'\n]));\n\nSo here, I've added a "democ" gulp task - shorthand for "demo my components". I pick up an argument from the command line and pass it through to Atom-Shell. Here's how I launch it from the CLI: "gulp democ --comp myawesomecomponent". When the argument value "myawesomecomponent" is passed, you can probably see that Atom-Shell is launched with an additional argument: components/myawesomecomponent/demo.html. Assuming you've been using bower or generally following Polymer's best practice component organization, this is the path to your component's demo file!
\nNext, we have to go inside our main.js file (Atom-Shell's entry point) and use some basic NodeJS code (nothing to do with Atom-Shell, really) to accept the argument and launch the appropriate page:
\n\n\n // Create the browser window.\n mainWindow = new BrowserWindow({width: 800, height: 800});\n\n // and load the index.html of the app.\n\n var startpage = 'index.html';\n if (process.argv.length == 3) {\n startpage = process.argv[2];\n }\n mainWindow.loadUrl('file://' + __dirname + '/' + startpage );\n\nSo here, we have some simple routing of our "startpage". If that argument is present, Atom-Shell will load a different HTML file that's not our application. In context with our Gulp task, we see that Atom-Shell loads our custom component demo page!
\nIt would appear that Atom-Shell works a bit differently than a browser. I'm not sure whether the fault lies on Polymer or Atom-Shell or what. In one instance in my application, I have "paper-tabs": basically just a tabbed navigation. Each tab can fire a click event. When receiving that click event in Chrome, I can look at the event.toElement.id to figure out which DOM element the click event is coming from.
\n\n\n <paper-tabs selected="0" flex style="max-width: 600px;" bottom indent noink>\n\n <paper-tab id="play_tab" on-click="">Play</paper-tab>\n <paper-tab id="library_tab" on-click="">Library</paper-tab>\n <paper-tab id="sources_tab" on-click="">Sources</paper-tab>\n <paper-tab id="tasks_tab" on-click="">Tasks</paper-tab>\n\n </paper-tabs>\n\n...the Javascript:\n\n onTabClick: function(event) {\n // note that parentElement is necessary for Atom-Shell, but is not used for web (dunno why)\n this.fire('tabclick', {id: event.toElement.parentElement.id});\n },\nSo what's odd here, is that in the browser, event.toElement.id will give me back an element ID of "play_tab" or any one of the other 3 tabs. In Atom-Shell, it will return "tabContainer" which is a child of my paper-tab and hidden inside the shadow dom. So in Atom-Shell, I need to look at the parent element, and get that ID.
\nNot too much trouble, but what other weirdness will Polymer bring to Atom-Shell? I don't know but so far, it's pretty awesome, and I'm glad I have a workflow down!
\nIn my haste, I neglected to remember that my "One Problem" above is not really a problem. I had completely forgotten that toElement is an obscure IE property. I had thought that it might be something new in Polymer. Instead, using event.currentTarget.id in my example works just fine. Interesting difference in behavior, but not one to worry about. Even better, even though it was a stupid issue, the Polymer team tracked it and closed it very quickly: https://github.com/Polymer/paper-tabs/issues/9#issuecomment-48226899
\n", "date_published": "2014-07-06T00:00:00Z" } , { "id": "https://benfarrell.com/blog/2014-05-12-creative-node-js-the-presentation/", "url": "https://benfarrell.com/blog/2014-05-12-creative-node-js-the-presentation/", "title": "Creative Node.js : The Presentation", "content_html": "In a couple days, I'll be giving the first stop on my "Creative Node.js" presentation tour! I'm speaking at cfObjective in Minneapolis on Wednesday May 14th. After that I'll be speaking at HTML5DevConf in San Francisco on Thursday May 22nd. I'm giving a talk about how folks can and should use Node.js for visual and creative work even though it's widely regarded as just another server side tech.
\nThe purpose of this blog post is to give some links you may see in my presentation. I'm also posting my slide deck as well for those of you who couldn't make it. The deck is checked into Github, and to see some simple code behind my examples, you can look here. Be careful with the OpenCV library, it's especially difficult to get installed, especially on Windows! NPM install doesn't cut it, and you need to manually build with nw-gyp, and add the required DLL's right inside the release folder.
\nA web hosted slide deck link really won't do it justice, though. As I worked on my presentation, it really became more of a desktop application than a slide deck. The presentation itself is powered by RevealJS. I've found that using Reveal, you can't place working script examples inside the page (presumably because of security concerns), so I have some links denoted by +'s to launch an example in a new windows. Some of these examples are AngularJS based, which is really nothing new.
\nWhat IS a little new, is that all of my examples except one require Node.js to run. I've actually put these examples directly inside my presentation by making the entire slide deck run by way of Node Webkit. I've written more about Node Webkit on a great site called Flippin' Awesome. So, what we have hear is a RevealJS slide deck running Node.js powered examples with some running with help from AngularJS, all running inside a desktop application powered by Node Webkit.
\nThis is why I love this platform - because we can put all of these technologies together and pull directly from the fabulous HTML/CSS/JS/Node community.
\nLeading up to Node.js, I mentioned Adobe Director, Adobe Flash, and a couple things you may have not heard of: OpenFrameworks and Cinder.
\nI also talk about a few of my projects. One, a music aggregator, playlist and radio show creator called SharkAttack. This project relies on an awesome video utility library called FFMPEG.
\nSecond project, was a Node.js based speech recognition library that uses either Google or AT&T services to translate text to speech. I called it "Make it So", because in my demos you pretend to be Captain Picard and tail each command with "Make it So".
\nI also listed a bunch of projects to give Node some kind of windowed view:
\nI really hadn't tried any of these in depth. I attempted Node-WebGL breifly, but had DLL and include file troubles, so shelved it in favor of Node Webkit and Atom-Shell which give you the whole webkit package. I talk about a quick sample Atom shell application called "WorkIt" for time tracking.
\nI also get into Bluetooth a little bit with a great Node.js addon from Sandeep Mistry called Noble. I also wrote a little Bluetooth library on top of that called "I Heart Node" to give a little easier and more understandable way to work with the TI Sensor Tag.
\nExpanding on devices, I talk about my Node.js/Midi/3D Sensor instrument called the Upright Spass powered by my 3D Sensor library called NuiMotion. And, with 3D sensing in mind, I also talk a bit about OpenCV for computer visualization.
\nAfter talking about all of these , I talk about making your own add on with an example gist. Even better, check out my old post on Node.js addon development.
\nLastly, I talk a little more about hardware with Johnny 5/Arduino, Node Bots, Node Copters, and running Node on a Raspberry Pi.
\nIf you missed my web/email addresses for working at GE Software in San Ramon, you can email here for jobs in our fantastic UX group. You can also look into our software center as a whole here.
\n", "date_published": "2014-05-12T00:00:00Z" } , { "id": "https://benfarrell.com/blog/2014-04-30-another-jerk-writes-about-google-glass/", "url": "https://benfarrell.com/blog/2014-04-30-another-jerk-writes-about-google-glass/", "title": "Another jerk writes about Google Glass", "content_html": "The last thing the internet needs is another jerk writing about Google Glass - yet here we are.
\nI've been pining to try Glass for awhile. I was, of course, jealous when my friend Adrian got Glass. Yet I wasn't willing to shell out the $1500 on a product that was still baking AND travel from North Carolina to NYC or SF to pick them up.
\nFast forward to August 15th, and I with a fresh tax refund in hand, a disapproving but not homicidal wife went over to http://www.google.com/glass/start/ and bought a set. Apparently they are sold out now, but for a short period of time anybody including myself could purchase.
\nI've heard about people getting attacked in San Francisco just for wearing Glass, so I was happy to see that accessory frames that Google sells on their site for$150 was available as a free add-on at the time of purchase. I knew it wouldn't make me NOT look like a complete jerk, but at most maybe 64% jerk. Somehow, at the time, I was also conveniently forgetting how I once felt about those who wore bluetooth earpieces as a fashion accessory.
\n![\"edge_ch\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2014/04/edge_ch.png\")
Anyway - I can tell you that the shades make things a little better socially. The first time I wore them, my wife who was engrossed in work, and barely acknowledging me saw the shades but didn't notice I was wearing Glass. So.........win?? More importantly, the sunglasses aspect allowed me to completely fabricate some social rules about wearing Glass. Since they are sunglasses, I'll only wear them outside while I'm active. In sunny weather, everyone's wearing things strapped to their face. It's normal. When I went inside, I'd put them on top of my head. Kinda rude to wear sunglasses inside, so I figure maybe its the same with Glass.
\n![\"ocularpatdown\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2014/04/ocularpatdown.jpg\")
What was the first thing I did when I got Glass all fired up? Tried everything of course. It ships with some games that get old really fast, but are pretty cool to show off the sensors on the device. Skeet shooting, for example, let you tilt your head to aim at the clay pigeons and then yell "Bang" to fire at them.
\n![\"googleglassminigames\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2014/04/googleglassminigames-300x165.png\")
Then I moved on to the translate feature. This was massively cool, though I was actually not aware that this app (WordLense) was already available on iOS and Android. Either way, looking at a French textbook that I stole from my wife, I could see the Francais fading away and the English overlaying it seamlessly.
\nI tried more apps and more apps. The wow factor didn't last long, so I decided to put them away and try them later in the field. This was also partially due to me trying to play some music, stumbling horribly with the UI, and eventually having a song play while overheating Glass and shutting it down. The UI is still a bit confusing to me, even though you eventually sort of remember how to use it. The main UI is a timeline of events. You swipe by running your finger along the side of the device and navigate to previous things that happened, like your last emails or phone calls. It was a bit hard for me to remember to tap the screen that says "OK Glass" to get to actionable things. Mainly, I think you're expected to say "OK Glass....google/call/text/etc" instead of tapping - but sometimes you just don't want to feel crazy talking to yourself, despite how awesome the voice recognition is.
\nSo, my first field test...
\nMy wife and I went away for a trip to the Santa Cruz Mountains for wine tasting. As my wife and I were enjoying the wineries, I definitely had my share of questions about Glass. Even one from a lady who asked if I was recording everything. This seems to be the biggest thing people are concerned about. Everyone: I assure you, I not only don't have the battery life nor the storage capacity, or a good reason to record everything I do. Taking pictures sure is easy though! If the option is on, you can enable photo taking by winking your right eye. This is a little problematic for me because I'm left handed and don't wink well with my right eye. I seriously have to move my entire face to accomplish this. Anyway, it's super great to take candid photos when the moment strikes.
\nI decided to load up some navigation for our car trip. This was awesome, actually. Personal directions right in your face instead of on a smartphone. I wasn't driving, but I could have been. This is the biggest revelation I've had for Glass - why it wasn't really what I expected at all. Glass is NOT a screen on your face. Instead, it's a tiny display above your line of sight....and it keeps going to sleep.
\nIn the fine print on the Glass documentation, Google describes Glass as a tool to enable microtransactions. I think that's the right word...it's hard to find, and I may have thought of the wrong term. The essence is that Glass is designed to use quickly and then ignore. There is even a setting that, when turned on, enables Glass to wake up when you are looking up at a certain angle. So, wearing Glass, you'll typically see nothing - but then when you want to check on something, you tilt your head upwards. After interacting for a few seconds, you stop what you're doing and the Glass display turns off.
\nNot only that, but you have to force your eyes to look up - above your normal sightline to see the display. Its actually kind of tiring if you constantly do it.
\nNow, both of these together - Glass just isn't a screen for your face. It's not augmented reality. How could it be for a screen that's painful to look at for a long time, constantly turns off, and is in a tiny window? The biggest surprise for me is that Glass is a wearable technology. As obvious as this statement is, because you wear it, you can set expectations similar to a smart watch. Small transactions where you want to see something quickly, and then you're done.
\nDoes this mean it's bad? No - but I'd say it doesn't have a real good use yet. It's nifty to see incoming emails, take phonecalls, and send text messages without a phone (in hand) - but it's just not worth the cost (moneywise or looking like a dork). Turn by turn directions are by far the best use. The Strava running app is a decent showing as well. Going on a walk/run/bike and being able to look up and see your speed, distance and time is excellent, but I think it needs to go a bit farther with things like heartrate and maps to make it all worthwhile.
\nI think Glass can really shine in tandem with a smartphone. Especially in the fitness world, or anything needing your hands free and your eyes on the road. Use your smartphone UI to set up an session in your app and then broadcast a heads-up display to Glass that you can always enable during your bike/run/walk/drive. Strava and the turn by turn directions take advantage of this, but really don't set themselves apart from a smartphone enough to make Glass worthwhile.
\nGlass is open country. There are so many use cases that hand held smartphones are just annoying and possibly dangerous in - where Glass could really shine. But, I just don't see the apps yet. As a developer, this excites me.
\n", "date_published": "2014-04-30T00:00:00Z" } , { "id": "https://benfarrell.com/blog/2014-03-25-hackfit-san-francisco/", "url": "https://benfarrell.com/blog/2014-03-25-hackfit-san-francisco/", "title": "Hackfit San Francisco", "content_html": "Yup, I'm pretty exhausted now because this weekend I attended Hackfit San Francisco. What is it? Well - it's a hackathon that focuses on fitness and health tech, but ALSO promises group exercise, reasonable bedtimes, and healthy meals.
\nI'd never been to a proper hackathon before, mostly because in my early career as a developer there were too many late night cramming sessions, and admittedly a few all-nighters. I've gotten my fill of that. I value my sleep and completely believe the notion that productivity drops like a rock when you're marathoning through a project from 10am to 4am the next morning.
\nThat's one of the reasons hackathons never appealed to me enough to convince me to go. There's also the fact that I love to hack, but the stuff I like to hack is HARD. More than can be done in a measly 48 hours hard, really. So the prospect of sitting for days on end with a stack of Mountain Dew + Pizza, freaking out about a deadline is just not appealing.
\nHackfit is not any of this. Late nights are frowned on. If anyone worked past 11pm, nobody admitted it. Exercise breaks happened often and actually factored into overall score for the weekend (THE reason we didn't place higher). Catering for lunch and dinner was done by Rebecca Jean Catering. I eat enough guilty pleasure food that I really don't go for "health" food so much. Thankfully it was really good. If the health aspect wasn't played up by Rebecca during her 15 minute presentation, I would have been none the wiser and just thought I was eating tasty, tasty food.
\nSo all in all, this event sounded like a good first hackathon for me. I've also been playing with all sorts of gadgets and movement related experiments in my spare time, so Hackfit seemed perfect. How did it actually go down?
\nWell, my first mistake was to ignore the Hackfit email. Hackfit sends you an email a week or so before the event introducing you to a "buddy". My wife would call it my "east coast demeanor"...but I would just say I just listened to my parents and don't talk to strangers. STRANGER DANGER! Am I right? Nevertheless, I never emailed my buddy and my buddy never emailed me. I would have resorted to full on wallflower mode if I was there on the opening night alone, but thankfully I was there with a co-worker. Even so, we were both on the fence for actually making it past the first night.
\nHonestly, it just seemed easier not to commit to a project, avoid the failures, exercise, early mornings, etc. But guess what? We kept meeting people.
\n\nRight off the bat, outside the event we met an awesome guy who flew in from North Carolina to come pitch his passion project - a group fitness app that brings people outside. Not only go outside, but give the fitness instructor a line to the class with a direct audio link. Awesome idea and I wanted him to be successful, especially because he flew across the country on a day's notice to attend. To be honest, though, I planned to stay far away from this idea. The heart of the app didn't seem do-able in a weekend. As a developer who's dealt with streaming media for 4 years, I knew I couldn't help because I knew how complicated these things can be.
\nThat, later, turned out to be team member #1.
\nRight before the pitches, another gal introduced herself to us. She was there alone - and obviously has a friendlier public demeanor than I, and isn't afraid to just walk up and introduce herself.
\nAnd there was team member #2 (as we'd find out later)
\nAt around 7pm or so, the pitches were made. Those who had an idea to pitch did so. Everyone did a fabulous job, but I was really only drawn to a couple of pitches. I was caught between not thinking the simpler ideas were fun/challenging enough to work on, but the complex ideas were TOO complex to do in the short amount of time.
\n(boo, this photo has been lost over time)
\nI was ready to cheer on the North Carolina guy we met as he was waiting in line to pitch, but another person pitched the SAME exact idea just moments before. He took it remarkably well, but I felt like I was watching America's Funniest Home Videos when the star of the video gets a football in the groin. I made the same contorted face and felt the pain. They both turned it into a win when they decided to team up. He and his girlfriend became team members #3 and #4.
\nThe pitches were drawn up on poster paper and displayed on the wall. Teams were forced to sell their ideas on an individual basis. They needed to gather signatures by people to form the groups. If teams didn't have a certain number of signatures in 10 minutes, the idea was scrapped.
\nBy this point - you know which team I was on. It was a hard decision though. I ultimately landed on peer pressure and just joined up with my co-worker and those who I already met. I'm also realizing that Hackfit did well. From meeting friendly people, to forced hugs during the intro speech, to sending the buddy email, to having teams pitch you one on one - Hackfit had masterfully done it. They orchestrated some pretty decent social connections to engage those that were on the fence like I was.
\nI should back up to before the pitches, though. I should highlight the point at which I realized that I overlooked something major. See, I'm sorta between fitness regimens right now. Which is to say - I've been lazy for several months. Despite me not being prepared in the slightest, FitMob was there to lead the first workout. After the workout, even relatively fit people were saying it was brutal. Me, I did most of it....not well...but I did it. I've worked with a personal trainer before, so I knew the points at which I needed to slow down and stop. I think Fitmob realized it too, because they are the sort of instructors that get up in your face to motivate you. I was probably white as a sheet by the middle and they let me take it easy.
\n(boo, this photo has been lost over time)
\nWe left on Friday night throwing out a few ideas, but adjourned not knowing what would come the next day.
\n\nAt least half the team arrived later (around 9 to 9:30) on Saturday morning. Between not being a morning person and having to take a fairly long BART ride out from the East Bay, I was one of them. I missed breakfast and the early morning workout session, but dove right into the work. As I was the only developer on the team, I felt some pressure to get something cool done. That's why I jumped right into what I thought would be a valuable tech demo for the Sunday pitch. I started right in as we were all throwing out ideas. Eventually in an hour or so, I felt comfortable enough with my trajectory that I filled in the rest of the team what I was plugging away on.
\nIf I did it again, I probably would have gotten more involved with initial planning. In hindsight, I may have railroaded the team into what I was doing. But it actually worked out really well. While I worked on the tech demo - the rest of the team worked out the business side of things. My co-worker was very involved in these convos, but broke off and started designing pure awesomeness in a short time.
\n(boo, this photo has been lost over time)
\nAs tech demos and designs and plans started shaping up, we all put our pencils down to head out to StudioMix for another workout session. I decided to bike the two miles from our workspace at Tagged. When I got outside, I realized that
\nLuckily Hackfit organizer Justin Mendelson (and another gal that I didn't catch her name) were leaving also on their bikes. I followed them, but being the out of shape bastard I was, had to take some San Francisco hills that just eventually killed me. Justin was kind enough to walk with me for short spurts.....I still felt a little bad though, especially because the hills didn't look THAT terrible.
\nAs Saturday wound down - we had fantastic progress going. I had most of the tech demo done with a few tweaks to be done the following day. I pulled heavily from some of my past experiments that I had posted on Github. I made sure to check with the organizers ahead of time to make sure this was cool. One of the rules was that hackfitters should not use pre-existing projects. I found out that a big problem with these kinds of hackathons was that people will jump from event to event pitching the same ideas and just trying to win and get investors. That's why Hackfit imposes the rule of making sure we actually create something from scratch during the weekend.
\nEvery so often, I glanced over at my co-worker Fernando's screen. Suddenly a logo. Suddenly screen mockups. Suddenly an interactive high resolution prototype. And it was all looking REALLY good. WTF. It's amazing what happens when you step away from the office and the established corporate brand and see what people you know can do.
\nWhile Fernando and I were plugging away, our other team member Allison, who just happens to be a product strategist at Frog Design was working with the ideas folks on business strategy and presentation. As I was a little disengaged from the group, I didn't exactly know what was shaping up - if they were actually doing good things or what. I just trusted them to do their thing and what happens, happens.
\n\nYet again, a modest 9:30 roll-in for most everyone. Again, most of us (if not all) skipped the 7:00 am Yoga session that turned out to be some sort of intense boot camp thing by surprise. Allison brilliantly brought her French Press to make coffee. Apparently in Hackfit-land we don't drink coffee. What's wrong with that zero calorie goodness? I can't fathom. Hackfit DID realize their terrible crime, and brought boxed coffee this last day.
\nI tweaked my demo a little most of the morning. I can't recall if Fernando did much this day, because I really don't know that he needed to do anything. We were both pretty low key and just helped the others with the pitch/presentation.
\nIt was time to present to everyone around 1pm. We decided to downplay the tech demo side and just really give me 20-30 seconds out of the 4 minutes. I suggested this because everyone had done such a fantastic job with the business plan, product, design, and application flow I felt that this should really be the focus. At the beginning on Saturday morning, I thought that if we did our 3 efforts separately (presentation, design, and tech) we could have 2 of them fail and still be mildly successful. Nope....all 3 were a slam dunk.
\nMy wife showed up right before our presentation to help out. We set her, Fernando, and Aaron loose to run around outside while we gave our presentation. Each of them were equipped with a smartphone loaded with our Phonegap/AngularJS app. The app tracked geolocation and sent it up to a Node.js backend. The backend would register each user and send the group's data back down to individual devices and the Macbook we were using to drive the tech demo. With this, we had simulated a "mission control" like display for an instructor to keep tabs on the group and display our markers using the Google Maps API. We didn't have a direct audio link, but we did have the next best thing with being able to track the crowd.
\nThe presentation went swimmingly, and the tech demo started a bit rocky but eventually our runners started showing up on screen around our area in San Francisco. When it started working, we got some great applause.
\n(boo, this photo has been lost over time) me giving the technical pitch and flailing my arms wildly
\nWe had high marks all around and made third place...a bronze medal for my first hackathon. I also dig the fact that we had the highest technical score out of anybody and slightly below the highest "wow factor". We lost in the "wow factor" category to a pair of gals that wowed me as well. I talked to them afterward. With no previous hardware experience, they made an Arduino controlled light up armband which reads your heartrate and calculated if your BPM was in the correct zone for working out. Amazing.
\nAnyway, that's the story of my team, the Wolfpack: an app that brings instructor led group fitness outside.
\nI really must thank the organizers of Hackfit for an exhausting, but fun weekend.
\n(boo, this photo has been lost over time) bronze medalists (my wife included cause she was an awesome runner during the demo)
\n", "date_published": "2014-03-25T00:00:00Z" } , { "id": "https://benfarrell.com/blog/2014-03-01-3d-sensing-state-of-the-nation-my-node-js-plugin/", "url": "https://benfarrell.com/blog/2014-03-01-3d-sensing-state-of-the-nation-my-node-js-plugin/", "title": "The 3D Sensing Apocalypse & My Node.js plugin", "content_html": "Dire straits are ahead for 3D sensing. I commented about it back in November - that I hoped Apple wouldn't kill open source 3D sensing library OpenNI from Primesense after the buyout.
\nIf you visit http://www.openni.org you'll notice that the entire website will shut down on April 23rd. Also unfortunate is that you'll no longer be able to download the OpenNI and NiTE package with drivers.
\nWorse, they are seem to be liquidating their stock of Carmine sensors. Emptying their warehouse means they probably will never make another one. Also, Asus seems to be no longer making 3D sensors. I can only guess that Microsoft will stop making the old-style Kinect soon when the new one for Windows dev comes out.
\nWhat does this mean?
\nWell...no more cross platform 3D sensor hardware and no more software to run it. Creative makes the Senz 3D which is powered by the Intel SDK and MS will be making the new Kinect for Windows. Both are Windows only.
\nThe cross platform market seems dead. Apple hasn't disclosed yet what they will do with Primesense, while Google has announced Project Tango: a device with a 3D sensor built in. Everyone seems to be going their separate ways with this tech.
\nProbably the best way forward is to get a used XBOX Kinect, download OpenNI/NiTE to a safe place before April 23rd, and try to get things installed with Freenect if you use Mac or Linux.
\nLuckily, I received a nice contribution to my Node.js OpenNI wrapper. Gordon Turner added some great documentation around making my Node plugin NuiMotion work on Ubuntu and OSX with the Kinect and Freenect. This gave me the kick in the pants to update the project to have a better build process and test/fix integration with the new OpenNI/NiTE binaries on OSX, Windows, and Ubuntu. I've also added an API to get raw depth or RGB data.
\nWith the right pre-install steps in place, all you'd have to do from any platform is a simple "npm install nuimotion".
\nSo, even though the crossplatform 3D sensing dream seems dead for now, we can stock up for the apocalypse and have fun until the next generation springs up. You can find NuiMotion at https://github.com/bengfarrell/nuimotion
\n", "date_published": "2014-03-01T00:00:00Z" } , { "id": "https://benfarrell.com/blog/2014-01-16-windows-for-front-end-web-development/", "url": "https://benfarrell.com/blog/2014-01-16-windows-for-front-end-web-development/", "title": "Windows for Front End Web Development", "content_html": "I've been using Ubuntu as my development environment for around 2 years now. Getting to know Linux has been surprisingly enjoyable - I had tried both Mint and Ubuntu, both of which were stellar OS's. I didn't switch from Windows back then because I expected a better OS, really. I did it because the web development world as we know it was using Apple's OSX en masse. To me, Apple has always been a little pricey for what it is. I also had a little bit of attitude at the time against Apple - it seemed to be taking over the world while creating it's own walled garden, and I wanted no part of that. Regardless of whether you have Apple hate, there was one thing that OSX had that was a big part of modern web development: the command line terminal.
\nNot having used the command line much since DOS days, I wanted to know what I was missing. To know the web dev world, I needed to immerse myself in the Linux command prompt that OSX had. And with my dislike of Apple, plus already owning a Windows laptop - I went the dual boot Windows/Ubuntu route. It was an awesome learning experience. With all of my dev work on the Ubuntu partition, I almost stopped using Windows entirely.
\nDespite how good Ubuntu is, I had a couple of nagging problems. The first was that my battery life was significantly shorter on Ubuntu. I have the feeling that it was because of the pre-installed Lenovo/Windows power management tools that were pre-installed on the Windows side of things. Second, and most important - was the lack of the Adobe Creative Suite. As much as I'd love to stop relying on our expensive industry standard tools, it was REALLY inconvenient to boot over to Windows every time I wanted to tweak a graphic.
\nI tried to get used to Gimp and Inkscape, but I really just couldn't. Graphics were a pretty big sticking point that disrupted my flow.
\nPast that, Ubuntu proved a pretty awesome environment for not only doing real work, but a good baseline to see what was REALLY cross platform. All of the tools we know and love in the web dev world work cross platform: WebStorm, Node.js, Git, Grunt, Bower, Yeoman, Sublime Text, Chrome, Firefox, etc. More significantly, while I was learning the Unix command line syntax - I was also learning some syntax that doesn't change from Windows to Linux to OSX. If you learn Git or Grunt on Linux, you've learned it on Windows as well.
\nThat was the key. If all we're doing is using cross platform command line utilities, then when it comes to OS choice, what's the diff? One of the main differences for people accustomed to Windows was that it really doesn't seem worth it to give the command line a chance. It seems counter to our nice GUI progress. I was certainly inclined to find visual based tools so as not to EVER touch the command line.
\nBut, after adopting Linux, the command line became second nature...it just felt like the right way to do certain things. So, when I got a new machine this past December - I was faced with a choice. Keep the copy of Windows 8.1 that it shipped with? Or dual boot Ubuntu again? Ultimately, I decided to see how much effort it was to rebuild my dev environment in Windows.
\nAll the usual suspects work as expected. WebStorm, Sublime, Node, Git....they all work great. Of course, as every Windows person can attest, the "command prompt" sucks. Luckily there are alternatives!
\nFirst I grabbed Console2. This is a great little replacement for the normal Windows Command prompt. Everything is configurable, just as good as the terminal from OSX or Linux. Though, what you type in the command line will be DOS based - so it takes some getting used to. No more "ls" to list directory contents - now it's "dir". But guess what? Install GOW (Gnu on Windows)! GOW will give you loads of the more commonly used Linux commands to use on Windows instead of the DOS syntax.
\nLots of stuff is covered now! My command prompt with Console2 and GOW is as nice as it ever was on Linux/OSX. I briefly installed Cygwin which has it's own command prompt and covers much more in the way of Linux, but unfortunately Node.js didn't like the unholy combo of running a virtual Linux on Windows (I think it was all line-ending characters that screwed it up).
\nMy favorite IDE looks just like it did before - except the terminal is now Windows based. But with GOW installed, it works pretty much the same.
\nAll in all, fantastic. My Windows setup seems just about as good as my Ubuntu setup was. Now, however, I can just boot up Photoshop for quick graphical edits. And when I want to play around with WebGL, the browser/OS support will be better.
\nWhat about just getting a Mac? Still a little overpriced for my taste. At my dayjob, I was given a MacBook Air. It's a nice machine - very tiny and comfortable in your lap. The battery turned out to be atrocious though. Barely lasts me 1.5 hours! The form factor did give me a taste for more with my Windows machine, though. That's why I bought a Lenovo Yoga 2. At around $1k, its close to the price of the Air. It's pretty much the same size, too. Some of the nice extras turned out to be a better battery life, and a screen that flips around back to be a tablet.
\nWhat really sets it apart for me, though, is the crazy good display resolution (3200x1800) and the fact that the screen is a touch screen. How handy is it if I'm working on a mobile version of my web application, and I can develop/test my touch events right on my dev machine itself?
\nAnyway, its quite nice to take my fear from 2 years ago that you really need a specific OS to do web development and debunk it. I've learned that if you use Windows, just make sure to TRY working outside of your comfort zone and get into the command line way of doing things. It really is worth it, and you'll set yourself up for some expertise that the rest of the community is already on to.
\nThis also proves one of the PC's main strengths over Apple - the fact that there are many companies trying out many different form factors. I don't need to wait for Apple to make me a Macbook Air with a Retina touchscreen - I can simply buy one from another manufacturer.
\nI should also disclose that now that I've been using Windows, OSX, and Ubuntu constantly - I like them all. Windows 8 has quirks, sure. I don't like how I keep accidentally swiping the edge of my trackpad and loading the Metro view. But then I have similar complaints for OSX and Ubuntu. That's the beauty of doing web development. I can just switch my OS again if I get sick of my current one.
\nHope that helps someone who might feel "stuck" on their PC and is wondering if they need to switch to Apple. Of course the answer is no, it's not necessary - but there's also certainly no reason not to give OSX or Ubuntu a chance. Try it all, and see what sticks for your personal taste!
\n", "date_published": "2014-01-16T00:00:00Z" } , { "id": "https://benfarrell.com/blog/2013-11-28-primesense-purchased-by-apple/", "url": "https://benfarrell.com/blog/2013-11-28-primesense-purchased-by-apple/", "title": "Primesense purchased by Apple", "content_html": "Anyone who's met me over the past year knows that I've been giddy over 3D sensing...you know - like the Kinect. I spoke at conferences about it and blogged heavily about it as I was learning the tech. It's been a fun ride.
\nIt was probably a year ago today when I first dug in to the OpenNI and NITE libraries from Primesense. Up until that point, I was experimenting with the Microsoft Kinect. The original Kinect was pretty awesome for it's time - but it just didn't sit right with me to have such a game changing technology in my hands and be tied so deeply to Windows. As someone who's career revolves around the web and multi-platform experiences it just felt like the wrong way to be.
\nThere are some technologies that will start fairly proprietary and end up in the public space. It's fairly obvious with Microsoft's track record that the Kinect wouldn't end up open sourced, or even multi-platform. You can't fault them for creating something awesome and keeping it to themselves, but it certainly leads to whole factions of people being left out of an important user interface movement.
\nThat's why I got so excited about Primesense. Historically, Israeli based Primesense was the original inventor of the 3D sensing technology and then sold usage rights to Microsoft to produce the Kinect. Primesense themselves created an open consortium/platform for 3D depth sensing software (OpenNI). Oddly, they kept the middleware (NITE) non-opensourced yet still downloadable. The middleware was the important bit that actually did the skeleton tracking, whereas the OpenNI project itself was really just a basic API to read 3D sensor data.
\nIn any event, we now have a semi-open platform for 3D sensing and skeleton tracking. Hardware to do it is extremely limited though, and I had hoped the situation would improve. You could either use the Kinect (Windows only), the Asus Xtion (no longer for sale I believe), or the Primesense Carmine.
\nThe Primesense Carmine was exciting, though. Here you had Primesense developing and selling the full Carmine as well as a mini version that was low powered and just poised to take over mobile. In fact, I had heard that Primesense was preparing batches of thousands (maybe even tens of thousands) for undisclosed device manufacturers. An open 3D sensing platform seemed likely to take over the world given a year or two.
\nThat was, until this week. Apple has just confirmed that it has purchased Primesense. It's no secret that I have no love for Apple, but hell, I wouldn't mind Apple getting a million or so orders of those awesome little Carmine units for upcoming iPhones. It woud be an awesome win for 3D sensing if that happened....
\nBut no - Apple now owns the only cross platform 3D sensor in town. I would assume that now we're back to a proprietary platform that only runs on one OS. Only, this might be worse than the Microsoft situation. Apple, in lots of minds, is increasingly becoming a consumer tech company. You often read Apple users complaining about Macbook Pro neglect, focusing on the iPhone and iPad. So now, presumably, we have 3D sensing technology in the hands of a company that will probably only ever deploy it on an iPhone, iPad, or Apple TV.
\nSo, going forward, I would assume that to make a Primesense 3D sensing UI experience we're going to need to develop in Xcode and deploy only to iOS which Apple will create ham-handed standards around and will block apps from the marketplace that don't conform. Again - great for consumers, but not for hobbyists, artists, and innovators - the very people that made this tech popular today.
\nYou might say I'm being overly doom and gloom. Could Apple simply keep up OpenNI/NITE and still sell Carmine sensors? Well, signs are not good. Right now the Primesense store has been shut down. You can't buy a Carmine anymore (will this change after the company changes hands?). And NITE? The skeleton tracking middleware? That was always a little suspicious, because it was never open sourced like OpenNI was. Perhaps it was secretive intellectual property that could be leveraged for a buyout. I don't see Apple doing good for the community and opening it up if Primesense didn't already.
\nI'm already speculating a little what Apple's goals are with 3D sensing. If strictly consumer, it would be a bit odd - since 3D sensors, being infrared based, don't work in direct sunlight. If you think about that and Apple's extreme attention to design detail so that things "just work", I just can't see a product like this being released as an iPhone or iPad under the Apple name with such a huge restriction given that lots of folks use their devices outside.
\nThat really just leaves the AppleTV, which is the perfect place for it - but an extremely limited market in comparison to Apple's mobile initiatives. It's sad when you think about it, because Primsense has featured tons of nifty companies and products that use their devices in all sorts of different environments and use cases. While some oddball tech company like Zenith or something will make an 80 inch flatscreen display with built in sensor, or someone else will come along and build a unique piece of hardware for a healthcare application - you'll never see Apple splintering it's product line like that. And of course, no other manufacturer is allowed to make these under the Apple umbrella.
\nContrast that with Windows. Small companies come in an make truly unique experiences. Many fail, but manufacturers try anyway. Natural User Interaction is still in it's infancy - and I truly believe that Apple won't allow it to grow naturally. They will lead and tell us what to do in regards to their own product line. Meanwhile, oddball, non-Apple like experiences be damned.
\nHopefully I am doom and gloom. I really dug what Primesense was doing. If they get to continue, it would be awesome. If not - well...the Kinect v2 looks pretty sweet!
\n", "date_published": "2013-11-28T00:00:00Z" } , { "id": "https://benfarrell.com/blog/2013-11-05-some-geolocation-and-google-mapping-services-in-angularjs/", "url": "https://benfarrell.com/blog/2013-11-05-some-geolocation-and-google-mapping-services-in-angularjs/", "title": "Some Geolocation and Google Mapping services in AngularJS", "content_html": "I've been having some fun with maps lately. It's been a little more involved than popping a marker on a map and showing a user where they are in a static way.
\nInstead - I need to do a fair bit of geo-math, animate a marker, and more. I picked AngularJS as my weapon of choice - so making some Angular services seemed to be the way to go!
\nFor reference, I'm working on a geolocation enabled virtual golf game I call "Geolf".
\n\nI won't go too far into detail on this first one - basically, I made a wrapper around the geolocation services provided in your browser. Beyond the "navigator.geolocation.watchPosition" which I've blogged about before, I added a subscription API. This was inspired a little by Java or C# (I can't remember which now) - basically instead of passing one and only one callback into the service, I keep a list of subscribers and fire off events to each one of those callbacks.
\nAngularJS has watch and observe methods, but I wasn't entirely sure if I could use them inside of services without an Angular "scope" to latch on to. Seemed more straightforward to write this myself just using callback arrays - and other services could tap into this service without a controller in the way. For example, my "mapping" service will want to know where you are, so it will subscribe to the "geotracker" service.
\nThere might be better ways to do this, but this worked for me - and it's really boring architectural stuff, and I'd rather talk about some awesome GEOMATH!
\n\nSo doing geolocation math is a little bit beyond me, so I turned to the internet which pointed me to the Haversine formula. The aforementioned page even has this stuff written in Javascript. Lazy me says "Yay!". What's a bit interesting, not having worked with geolocation math before, is that the latitute and longitude measurements need to be converted into radians. I guess it's unsurprising from a trigonometry standpoint, but for me, not even knowing what latitude and longitude REALLY represent - I was clueless.
\nAlso interesting is the fact that you need to take into consideration a constant which represents the earth's radius. At 6,371 kilometers, when I write this up as meters - I'll of course use 6,371,000 meters.
\nHere's an example of using the Haversine formula to calculate the distance (in meters) between two lat/long pairs:
\nthis.calculateDistance = function(geo1, geo2) {\nself.convertFromGoogle([geo1, geo2]);\nvar dLat = self.toRad(geo1.latitude - geo2.latitude);\nvar dLon = self.toRad(geo1.longitude - geo2.longitude);\nvar lat1 = self.toRad(geo2.latitude);\nvar lat2 = self.toRad(geo1.latitude);\nvar a = Math.sin(dLat/2) * Math.sin(dLat/2) +\nMath.sin(dLon/2) * Math.sin(dLon/2) * Math.cos(lat1) * Math.cos(lat2);\nvar c = 2 * Math.atan2(Math.sqrt(a), Math.sqrt(1-a));\nreturn parseInt(R * c);\n}\nThere are two convenience functions here that I have elsewhere in my service. One is to convert to radians, and the other to convert a Google Location object to latitude and longitude. Google has a fair bit of structure around their lat and long, and we're just pulling out what we need.
\nLuckily, the nice rundown of the Haversine formula contains all the explanation I need to get some other methods up and running like getting the bearing between two locations, and getting the location when you project out from a location at a certain distance and bearing.
\nI've wrapped them all up into a nice Angular service here.
\n\nA Google Maps AngularJS service? For my application purposes, yes. Though it is fairly "singleton", so it might not work best for you. It works for me because I can have one map per application page and do some stuff to it. Some things I wanted to do was add custom markers (one that even uses an SVG path!), move the markers on the screen, move the view/camera, and even animate all of those.
\nThe animation of the camera and markers were pretty interesting. I initially used the browser's "requestAnimationFrame" to iterate my frames and move the marker frame by frame. However, I also wanted this to work in Phonegap/Cordova. Little did I know that "requestAnimationFrame" isn't supported there, so I also implemented an alternate setInterval animation routine. I offloaded the animation to a service of it's own and had each frame callback to my mapping service to cause something to happen on the map, whether it's changing the position or size of the marker.
\n\nOne of the last things I'll mention is orientation - but only because it was so hard to spin a compass in my Google map. Here I'm using Phonegap/Cordova to send me the magnetic heading of the compass. The call itself is pretty similar to getting your location - so the service is very similar to my geotracker service. I've only really added the ability to set the heading manually so I can test this on my laptop instead of my Phonegap/Cordova app.
\nAnyway - the tricky part to this all was updating my SVG drawn compass needle in Google maps. When creating a marker in Google Maps like this, you can set the rotation along with the rest of the properties like so:
\nnew google.maps.Marker({\n map: self.map,\n position: latlng,\n flat: true,\n rotation: 24\n icon: {\n anchor: new google.maps.Point(168, 900),\n path: "M 337.14285,951.79076 C 336.85134,1085.9601 1.8926143,1088.4865 -5.7142858e-6,951.79076 -5.7142858e-6,827.13221 169.31441,188.15081 168.57142,198.07649 c 0,0 168.57143,629.05572 168.57143,753.71427 z",\n fillColor: '#333',\n fillOpacity: 1,\n strokeColor: '',\n strokeWeight: 0,\n scale:.035\n }\n });\nNote that "rotation" property. Easy right? Now what about updating the rotation of the marker inside Google Maps as we get new orientation events when we point our phone south instead of west? Well, not so easy.
\nThere actually doesn't seem to be a way to do "setRotation" on Google maps for a marker or icon. You can drill into the object itself, and set the rotation property, but that doesn't update the graphic. So you end up having to almost create a brand new icon, but not quite like so:
\nvar ico = $scope.player.marker.getIcon();\nico.rotation = heading.magneticHeading;\n$scope.player.marker.setIcon(ico);\nAs you can see here, we're getting the icon, setting the rotation, and then setting the marker icon again with the new rotation. Not the direct most approach - but seems to work!
\n\nYah, I know - lots of pretty specific problems to my game/application. So maybe this code isn't super portable for you. But there is a lot of neat tidbits in here that I learned along the way. I thought that the geo math would be more difficult, but the Haversine formula seems pretty well established and easy to Google.
\nGoogle Maps are a world of their own - the API seems very geared towards Java devs using GWT, but at the same time pretty well documented until you hit some weird use case like wanting to rotate your marker.
\nThe big takeaway for me is that Angular services seemed a really good pattern to follow for this mostly-sorta-generic stuff I was doing. I can unload from my controller, and so long as it fits a "singleton" design pattern, it really works. One more notch in the Angular belt - I maybe went a little crazy on services here, but they fit what I needed in my application.
\n", "date_published": "2013-11-05T00:00:00Z" } , { "id": "https://benfarrell.com/blog/2013-10-15-its-the-open-sourced-sharkattack-yeah/", "url": "https://benfarrell.com/blog/2013-10-15-its-the-open-sourced-sharkattack-yeah/", "title": "It's the Open Sourced SharkAttack, yeah!", "content_html": "I've been working on my project called "SharkAttack" for awhile. It started out as a simple little Adobe AIR app to download new music from RSS feeds. Soon after, I rewrote the whole thing as a service with Node.js. And then, just lately I rewrote some of THAT to use Grunt.js.
\nAnyway, the aim of SharkAttack is to start with a list of your favorite music news feeds that will occasionally offer free music. The service will then dive into these rough seas and download the latest tunes a few times a day. It will deal with RSS feeds, webpages, YouTube, Soundcloud, Vimeo, and plain old MP3.
\nSharkAttack will convert, transcode, and track the songs. It publishes a playlist for both my personal website http://play.blastanova.com and for my radio show every Wednesday at 1pm EST on the awesome http://www.codebassradio.net under the leadership of the even awesomer Vicky Ryder.
\nIn the past few weeks, I authored some admin pages to add/remove/edit your sources, review logs, and edit my weekly show script. Additionally, I added some personal radio functionality - meaning that you open up a page on your local network and the server will stream a personal radio station based on the songs that SharkAttack has for you at that moment.
\nI'm at version 1.9 right now. As I approach 2.0, I'd like to get the streaming solid (right now it's a bit iffy) and get some really personal stuff in there like reading of your Twitter/Facebook messages as "radio ads" between songs.
\nIt's great to take a project like this all the way to something I enjoy using (or listening to). It never would have happened without my weekly radio slot on Codebass to keep making the SharkAttack better. And now I think it's got a few additional purposes as I keep adding functionality.
\nHere's a rundown of the tech I used:
\nAnyway - it's out there on GitHub land now. It's pretty fine tuned for my purposes, but maybe it'll help someone else, either as a whole or with the code torn apart in bits.
\nhttps://github.com/bengfarrell/sharkattack
\n", "date_published": "2013-10-15T00:00:00Z" } , { "id": "https://benfarrell.com/blog/2013-09-30-a-supervisor-for-node-js/", "url": "https://benfarrell.com/blog/2013-09-30-a-supervisor-for-node-js/", "title": "A Supervisor for Node.js", "content_html": "So, I'm writing this blog post because, despite how simple a concept the "Supervisor" is - I feel like I could have easily missed this super handy tool. In fact, I might go so far as to say that if I hadn't had co-workers push the Python supervisor on me - I would have had a problem I just wouldn't quite know how to solve.
\nLast February, I started learning Python because I had some downtime at work (front end dev wise) and I had an interest to get into the server side tool all of our devs were using. It was a fair bit to take in...Python was a new and fairly different language than what I was used to. It turned out fairly enjoyable.
\nAnyway - there was lots of stuff thrown at me, making CentOS VMs, administering Unix, all that stuff. You know, that stuff that doesn't have much to do with programming - but the environment/platform itself. In the midst of running learning how we have our own package manager in company for locking down Python Eggs, the "Supervisor" was thrown at me out of the blue.
\nAt first I thought it was some half cocked idea our developers put together in-house. What is this thing? Why do I need it? I just want to code some Python! I truly wanted to ignore it, because I never had a good idea what it was setting out to accomplish - plus it had some separate config files...just ick.
\nIn the end though - I finally understood it. Once I got past the code, and got things into production - only then was the time it was really needed. See...a supervisor process will run alongside your code or server. If it goes down, due to an error you didn't forsee, supervisor keeps tabs on it, and will just start it up again. Just log the error for viewing later - with the important thing being that your server or your process stays up.
\nMonths later, I found myself with my first Node.js/Express webserver project that I wanted to make sure it stayed up. Yes - I could make DAMN sure to test, test, and test. Make sure no errors happened to bring down the server. Despite my resolution of no fatal errors (yeah, right) - I looked into the Python equivalent of supervisor for Node.js. Yep - I found one. It's called Node-Supervisor.
\nSimply do a global NPM install and then start your project with it:
\nnpm install supervisor -g\nsupervisor myapp.js\nFrom that, we have a nice little process that monitors your Node.js app. If it goes down due to fatal errors, your script will get started right back up. The HUGE bonus to this is that I'm always iterating my project. My version number will keep going up as I add more to my project. That's the other benefit here....If I do a git pull, then a checkout of my latest branch/tag, supervisor will restart my Node app with the latest version. No hunting and killing the process, no restarting manually. Just pull and checkout, and the latest is instantly running.
\nAll thanks to Supervisor.
\nAgain - feels simple. Feels like everyone in the world already probably knows this. But me, not being a fully focused server side dev - well I might have easily missed this. A problem I might not have known I had, with a solution I wasn't really looking for. But there it is...Supervisor is pretty sweet!
\n", "date_published": "2013-09-30T00:00:00Z" } , { "id": "https://benfarrell.com/blog/2013-08-08-fun-with-geolocation-and-the-foursquare-api/", "url": "https://benfarrell.com/blog/2013-08-08-fun-with-geolocation-and-the-foursquare-api/", "title": "Fun with Geolocation and the FourSquare API", "content_html": "I'm on a race to get some blogging done before my lovely wife and I embark on a weeklong car trip to move to California. Sure I can do some on the road, but I definitely want some screenshots for this post, so here I go!
\nI'm been mixed up with 3D sensing all year - speaking at conferences, writing a Node.js plugin, and doing some interaction experiments. My weapon of choice has been my handy Asus Xtion Pro Live, and I just got a Leap Motion.
\nNet effect - I know where my fingers and body parts are at all times.
\nThe thing is, we all have something similar in our pockets for body tracking - it's a little less exact, but it's also almost universally available to those of us with smartphones. Of course, I'm talking about the GPS. If you can imagine interaction on a much larger scale than in your living room, you can make some pretty fun interactions using what is essentially a 3D sensing mechanism that tracks a single point...you!
\nI had done some Android/Java based experiments previously - and in all honesty, they are much like what you get with your web browser and Javascript. That's probably because the hardware itself and hardware API has pretty specific set of touchpoints.
\nYou'd think it were simpler, but the options are there to get as specific as you need while trying to conserve your user's precious battery. I had made the initial mistake in my application to use navigator.geolocation.getCurrentPosition() and running that on a timer not realizing that there is a navigator.geolocation.watchPosition(). The "getCurrentPosition" according to the Firefox docs is quick and lazy. It is supposed to get you the current position as fast as possible without worrying too much about accuracy.
\nThis is great for doing something quick to get a single location to tell where you are once when you load a map. Your phone PROBABLY has some reasonable approximation of where you are in the last several minutes from either your wifi or GPS - and it can pinpoint you on a map.
\nBoth set and watch provide 3 parameters: the success callback, an error callback and options.
\nFor me, I wanted to watch the user as they walk around with a high degree of accuracy. So I used watch.
\n\n var geo_options = {\n enableHighAccuracy: true,\n maximumAge: 5000\n };\n\n navigator.geolocation.watchPosition(mysuccesscallback, myerrorcallback, geo_options);\nThe options were key as well - I'm asking for a high degree of accuracy, and I'm telling it to discount any location results greater than 5 seconds old. YES - this will suck up your battery, be forewarned.
\n\nOK, so....we get a few things back in our "mysuccescallback". Most importantly, we get a coordinates object with latitude and longitude. We also get an accuracy number in our coordinates (to tell you how many meters your results might be fudged). I haven't worked with these numbers - but I do believe we get a heading and elevation number as well.
\nNow, latitude and longitude numbers are cool and all, but I'd like to know what's around me as well. Any interesting locations?
\nWe can make a call to the FourSquare API with https://api.foursquare.com/v2/venues/search?ll=<your latitude>,<your longitude>&oauth\\_token=<your auth token>&v=<a version date>
So, above, latitude and longitude are easy enough to surmise (we just talked about them)? What about an auth token? What about the version date?
\nSadly enough, Foursquare isn't COMPLETELY open for everyone to just use willy-nilly. You need to sign into your account as a developer and make an app:
\n![\"foursquare\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2013/08/foursquare.jpg\")
When you make an app, it will give you a client ID and secret (no you can't have my secret). Also, very important is to include any and all redirect URLs that your project will go to actually use the service for your app.
\nBasically, what you might do is to create a link that the user will click on:
\nhttps://foursquare.com/oauth2/authorize?client_id=ES4I1GUG5B5JZ5JNMNVZYVOU0X3QX5TZNQV0W34VDREXXEKS&response_type=token&redirect_uri=http://localhost/projects/finalfoursquare-ui/src/game.html
\nWhen the user clicks the link, they will be taken to a FourSquare page that allows them to authorize this app to use their account on their behalf. If they accept, they will be redirected back to the page you specify. In my case it's game.html on my local server.
\nWhen they get authorized - they get an auth token. Finally we get back to the call we originally wanted to do:
\nhttps://api.foursquare.com/v2/venues/search?ll=<your latitude>,<your longitude>&oauth\\_token=**<your auth token>**&v=<a version date>
Your auth token will appear in your address bar after your redirect page:
\ngame.html#access_token=XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
\nI just pick it up with the following Javascript:
\nvar my token = window.location.hash.substr(window.location.hash.indexOf("=")+1, window.location.hash.length);\nThe <a version date> thing I mentioned is so that FourSquare can change their API behavior, and it won't break your project. Supplying a version date will specify the date you made your code and use the FourSquare API as it appeared on that date. I started my project in June 2013, so I'm using "20130601".
Making this call will return a nice JSON response of nearby locations, what kinds of locations they are, their exact lat/long and much more. You can go ahead and parse this however you see fit. I did and here's what I'm starting to make:
\n\nPlease note that FourSquare does NOT allow the name FourSquare in your app name, but nothing about this is final. Right now it's appropriately named because I've been playing a lot of Final Fantasy III on my Nintendo DS. I thought it would be awesome to make a similar style of game where you wander around in the real world instead of a virtual map.
\n![\"ffsq\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2013/08/ffsq.jpg\")
Here I am wandering around the real/virtual world. Note that the accuracy in this screenshot is awful (within 43km) due to the fact that I'm on a desktop computer in a company's wired network. If I'm on my phone, my position can be detected within a few meters.
\nBut hey, I can still fight a local bank (and win - in one blow)
\n![\"fight\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2013/08/fight.jpg\")
The rough graphics here don't do it justice - I have a little spinny CSS3 transform as the characters fight. The hit points quickly rise and animate as the characters are struck.
\nLots of nifty potential here, and I've been doing some play testing to see what works and what doesn't. This is just one of the interesting and interactive ways we can use our position and locations around us. I have a few more ideas as well. I should also note that I purposely left out my dealings with Google Maps. Frankly its awesome to work with, but its a fairly big subject, and I'm not nearly done exploring it yet!
\nFrom a tools perspective - I've built this prototype with Node.js and MongoDB for mapping data to game objects and saving the player state. On the UI I've used AngularJS to layout the game and do the application logic. I've also used GruntJS and LESS for doing builds and updating my website.
\nI won't open this up for demos yet though - too many changes coming as I play and figure what works and what doesn't.
\n", "date_published": "2013-08-08T00:00:00Z" } , { "id": "https://benfarrell.com/blog/2013-08-07-interaction-zones-and-the-leap-motion/", "url": "https://benfarrell.com/blog/2013-08-07-interaction-zones-and-the-leap-motion/", "title": "Interaction Zones and the Leap Motion", "content_html": "I finally got myself a Leap Motion! It came in the mail around last week and I promptly plugged it in to play around. Prior to getting it in the mail, I read some great reviews of the hardware, but many frustrations around interaction.
\nAfter using it, I definitely agree, but I can also elaborate...
\n\nThe Leap Motion controller is a little white bar - almost the size of a bulky pack of gum
\n![\"what1\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2013/08/what1_desktop-9ccf2bb953da97fb3e713dbd47338561-e1375897900508.jpg\")
When I opened the box, I saw 2 USB cords. After my initial "why would they waste effort to ship two cords" thoughts and using it - it was much appreciated! One is short and one is long. When I place the Leap next to my laptop, I don't want a lot of extra cord hanging around, but on my desktop, it needed to reach floor level from my desk.
\nIt just goes to show that I think the attention to detail they put into this hardware was pretty good! Loading up some sample interactive demos was also pretty impressive. You can watch all of your fingers track quite well - it's really impressive to see the each finger tracked in a very responsive way.
\nAll in all - the hardware was super impressive, especially for such a tiny package.
\n\nNow - the interaction. Well, the out of box experience leaves much to be desired. Honestly, if you're not a developer with an imagination, I'm not sure what point I see to purchasing this yet. If you have little imagination and have just purchased this to be a mouse replacement - you're going to have an awful time. But, if you're a developer who wants to create unique interactions - it is definitely appealing. The "Cut the Rope" game, I thought was a pretty decent example of a unique interaction (swiping your fingers to cut a rope) that is immediately accessible to the user.
\nThere are 2 shining examples of why this isn't a showcase device for mouse replacement - and it all comes down to a simple touchless sensing interaction mechanism.
\n\nIt seems fairly obvious to use "touch zones". If your hand is far back, you "hover" - but if its moved farther forward you are "touching" or interacting with the screen.
\nThe "Touchless" application provides a good screen to explain:
\n![\"slide2-1\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2013/08/slide2-1-e1375898666547.png\")
At first, this seems perfectly reasonable. However, it becomes insanely difficult to use for many tasks.
\nI fired up Corel's beta application for painting. If you aren't familiar with Corel, they make a mean alternative to many Adobe tools like Photoshop, Illustrator, etc. I applaud Corel for trying something like this out!
\nInitially it was fun to use, but actually getting a feel for drawing was difficult. They had some gestures baked in that I haven't really had a chance to get acquainted with yet - and I think those will be key (I remember trying to pick a color and it wasn't VERY intuitive, but I thought it had potential). What I mainly focused on was drawing. Pen to paper.
\nCorel used the hover/touch zones as well - and it was just annoying. Firstly, you can put the Leap Motion anywhere you want on your desk. If it moves, you're constantly figuring out where that boundary between hover and touch is. Its nearly impossible to remember - especially because when we move our hands in and out, we're not doing it in a perfectly straight line. We're introducing a little arc as our hands rotate around our elbow.
\nI try to play a little keyboard/piano in my spare time. I'm not bad - and a basic thing when using this intricate and complex user interaction device is both tactile response and muscle memory. When I hit a key on my piano, I can feel it. My hand knows when to stop, and when to apply more pressure to make it louder. When I play a piece, my fingers remember where they were before if I practice enough. I don't have to keep reading the sheet music - instead my "muscle memory" takes over. This means that I know what the song feels like to my fingers when I play it, and I can learn how my muscles feel when things are going well...and repeat it.
\nCompare that with painting in air. Your Leap Motion might change position or you might shift in your seat and use a different hand position. There's really no muscle memory here to allow you to remember how you interacted with the device. And with no tactile response from the air, you don't get feedback of your pen hitting the paper. So for me, it was constant guess work on how to pull my pen up from the paper and not make tons of erroneous strokes.
\nThe "Touchless" application was even worse. It runs as a background task, and allows you to replace your mouse with the Leap Motion. Again - we're using hover/touch zones. Still the same problems as painting, and as a result it's a little hard to click on things - and almost impossible to double click. Moving your finger straight in to click, moving it back out of the touch zone, and then moving it back into the same exact position to perform the double click is incredibly hard!
\nSo - imaganitive games and experiences like "Cut the Rope" - WIN. Replacement for existing two dimensional interaction methods? I hate to say it, but fail.
\nI was a little sad honestly - the mouse and paintbrush seem the most obvious applications to go and replace. I started getting it in my head that we might never replace them with 3D sensing well, as they are inherently 2D interaction. In my past work playing around with the Kinect and other 3D sensing devices, I had a similar problem. They are quite similar things despite one being for your fingers and the other being for your whole body.
\nWith whole body sensing, I had the idea to use the angle of your arm to figure out how far you've extended your arm. There is a somewhat tactile response as you feel your elbow lock straight, which aids in muscle memory for doing the action as well. Doing an action feels the same every time because it's relative to YOUR body.
\nNow, I don't think you can make similar things happen with your hands with the Leap. For one, I don't think that any of the joints are captured - only the finger tips. And from what I saw, your fingers have to be reasonably outstretched to be picked up (and not balled up in a fist).
\nBut what about relative finger placement? When I do finally get around to using the API (Javascript and browser enabled, woo!) I'd love to try some interaction mechanisms like this. What's more obvious and natural than using your pointing finger to paint? Well, duh....we hold brushes and pencils with 2 fingers - our pointer finger and our thumb. Perhaps the proximity of these fingers could dictate if we are "touching" or "hovering". Can we click by tapping the pads of our fingers together? I think there might be some difficulty in sensing when fingers are so close like this, but definitely something to at least try!
\n\nBottom line is, have fun developing for the Leap Motion. Its worth it for you if you want to make cool things - we need to make it worth the common people to actually use!
\n", "date_published": "2013-08-07T00:00:00Z" } , { "id": "https://benfarrell.com/blog/2013-07-25-gruntjs-for-those-with-shit-to-do/", "url": "https://benfarrell.com/blog/2013-07-25-gruntjs-for-those-with-shit-to-do/", "title": "GruntJS for those with shit to do", "content_html": "Look man, I just want to use Grunt. I got shit to do. I don't want this to be a big thing - I just want to get in, set up my tasks, get them running, and get out and continue on with the REAL work in my project.
\nUnfortunately - the GruntJS project page does not help those with my mindset. It gives a quick overview then points you to a painfully slow step by step process. That's great for some folks. Maybe i'd like to spend an afternoon meticulously following along and playing connect the dots. It can really help some people.
\nNot me, I got shit to do.
\nTo be clear, I'm really not bashing on the project or the documentation. Its a fantastic project, and the documentation I'm sure is great. It's just not for me (the docs).
\nLet me lay down what GruntJS is first. They call it a "task runner". In lay terms - it means it will run a series of tasks for you when you type "grunt" into the command line. These tasks could be anything from minifying Javascript or CSS, or even copying or FTPing some files somewhere. People are writing Grunt modules for most everything you could want. Most web development things you'd probably want are most likely already written. And guess what - if you're a JS developer, you can write your own to fill in the missing pieces.
\nWhen I learn something like this because I want to get in and get out - I have a logical set of questions in my head.
\n\nGrunt is actually a Node.js project. We grab the Grunt project from the Node Package Manager (NPM). We install it globally by specifying the -g option when doing the npm install.
\nnpm install -g grunt-cli\nIf you have no clue what I was just talking about and don't have Node.js installed or NPM - look into that. I won't cover that here - but it is a fact of life that you'll need both of those installed to continue. Don't worry - it's not difficult!
\nSo after we install it, we can run it in any directory we want by simply typing "grunt" on the command line. But will it do anything? Next question!
\n\nWhen you run Grunt, it assumes there is a "GruntFile.js" file in the directory where you are when you typed "grunt". Don't have one? Make it! Right now. Here's some good docs on the GruntJS project page on how to make one: http://gruntjs.com/sample-gruntfile.
\nHere's my sample which minifies some JS, runs LESS, copies my files to a build directory, and FTPs them to my site:
\nmodule.exports = function(grunt) {\n\n // Project configuration.\n grunt.initConfig({\n pkg: grunt.file.readJSON('package.json'),\n uglify: {\n options: {\n banner: '/*! <%= pkg.name %> <%= grunt.template.today("yyyy-mm-dd") %> */\\n'\n },\n my_target: {\n files: {\n 'build/<%= pkg.name %>.min.js': [\n 'src/js/angular.min.js',\n 'src/js/directives.js',\n 'src/js/services/Library.js',\n 'src/js/services/WebServices.js',\n 'src/js/services/AnimationServices.js',\n 'src/js/services/Formatting.js',\n 'src/js/game/Characters.js',\n 'src/js/game/CharacterUtils.js',\n 'src/js/game/Fight.js',\n 'src/js/game/Items.js',\n 'src/js/services/GeoService.js',\n 'src/js/controllers/GameController.js',\n 'src/js/prod_cfg.js']\n }\n }\n },\n less: {\n development: {\n options: {\n paths: ["style"]\n },\n files: {\n "build/<%= pkg.name %>.css": "src/main.less"\n }\n },\n production: {\n options: {\n paths: ["style"],\n yuicompress: true\n },\n files: {\n "build/<%= pkg.name %>.css": "src/main.less"\n }\n }\n },\n copy: {\n main: {\n files: [\n {expand: true, cwd: 'src/assets/', src: ['**'], dest: 'build/assets'},\n ]\n }\n },\n\n 'ftp-deploy': {\n build: {\n auth: {\n host: 'ftp.mysite.com',\n port: 21,\n authKey: 'key1'\n },\n src: 'build',\n dest: '/htdocs/mysecretwebdirectory',\n exclusions: []\n }\n }\n });\n\n grunt.loadNpmTasks('grunt-contrib-uglify');\n grunt.loadNpmTasks('grunt-contrib-less');\n grunt.loadNpmTasks('grunt-contrib-copy');\n grunt.loadNpmTasks('grunt-ftp-deploy');\n\n grunt.registerTask('default', ['uglify', 'less', 'copy', 'ftp-deploy']);\n\n};\nNow lots of this is pretty custom to my project - like maybe you don't want to run a task to copy stuff, or ftp stuff - it's all up to you. There are 3 major pieces here though (working backwards):
\n1. Register task - this is the task list in the order you want stuff done
\n2. Load NPM Tasks - this is where you tell the task runner what tasks your masterpiece is dependent on. How would Grunt know how to run "uglify" if you didn't tell it what project uglify actually is or where it finds the code?
\n3. Init the Tasks - this is pretty custom to each "Grunt module". My uglify module has specific ways to tell it what to do. When I learn the secret ways - I toss it into that init section. To obtain the recipe, just do a google search for each project you want to use. In this case: "grunt-contrib-uglify" and copy/tweak an example or two.
\nNow, about step #2 - you didn't really think it was that easy did you? Just because we have a name (eg: grunt-contrib-uglify), how does it point to the Node module?
\n\nEasy if you know Node.js! Use a package.json file. See, when you're working with Node.js, there might be some dependencies. Maybe there's this really cool module out on NPM (Node Package Manager) in the "cloud" that ftp's files for you. You don't need to know how FTP works, you just gotta download the NPM module and include it as a dependency for use in your Node.js project.
\nTo install a module - you'd usually type in the command line:
\nnpm install myreallycoolftpmodule\nCan you imagine though, if you had a hundred developers and told them each to install 19 dependencies for your masterpiece of a project? We need a way to automate things - and make each developer run something to install ALL THE REQUIRED pieces or dependencies for our specific project so that nobody screws up.
\nThat's why if you type in the command line:
\nnpm install\n...it will use the package.json file in that directory that you're in. So a package.json file is a manifest of everything your project is dependent on. Running npm install, will sort through your list of dependencies, download them, and copy them into a "node_modules" folder in the root of your project. That way, when your project runs - it will have everything it needs.
\nHere's my package.json file:
\n{\n "name": "ffsq",\n "version": "0.1.0",\n "devDependencies": {\n "grunt": "~0.4.1",\n "grunt-contrib-jshint": "~0.6.0",\n "grunt-contrib-uglify": "~0.2.2",\n "grunt-contrib-sass": "",\n "grunt-contrib-less": "",\n "grunt-contrib-copy": "",\n "grunt-preprocess": "",\n "grunt-ftp-deploy": "",\n "grunt-env": ""\n }\n}\n\nAnd oops - I got some more dependencies in there than I'm actually using on this project - but you get the point. Also, if I don't provide the version number, I'm throwing caution to the wind and saying: "Just give me the latest, I don't care what you broke when you updated".
\nOK you've done all the prep work. You've created a package.json in your project root to let Grunt know what modules you need. You've created a GruntFile.json file to initialize everything and say HOW to run your tasks.
\nBefore you run it. Do an NPM install. This will actually install everything from package.json so Grunt can work.
\nThen just type grunt in your command line - and you'll see good stuff happen, OR bad stuff happen. If bad stuff - you can check out the output and see what went wrong. If you want crazy details, run "grunt -verbose". Doing verbose mode will really help if you have something that doesn't work and you need to troubleshoot.
\nThe hardest part of this will be learning the secret ways of the bazillion individual modules you employ in your project!
\nGood luck!
\n", "date_published": "2013-07-25T00:00:00Z" } , { "id": "https://benfarrell.com/blog/2013-07-23-you-suck-ncdevcon/", "url": "https://benfarrell.com/blog/2013-07-23-you-suck-ncdevcon/", "title": "You Suck, NCDevCon", "content_html": "Yah, I mean it...I can't go and you suck for being awesome. And you suck even more for being even more awesome this year.
\nNCDevCon has a special place in my heart. I was there for the first one, actually I was there for the negative 2nd one back when it was CF in NC. I attended a planning meeting or two. I did their logos. Most importantly, they gave me a shot to speak at a conference! I'd never done that before, and I'm sure it showed. But, I kept coming back year after year - and I got better.
\nLast year, for 2012 I gave what would morph into my "Sweatin to the Web" presentation that I gave this past spring. I dressed up in Richard Simmons'esque workout garb and talked about motion and the web.
\n![\"sweatin\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2013/07/sweatin.jpg\")
Would I have done this if it wasn't a fun, laid back, yet professional conference? Nope - I'd have chickened out I'm sure. I would have never graced the audience with my pale white thighs as I wore my short shorts for sure.
\nThe NCDevCon organizers are laid back - but what really set precedence is when they toilet papered Mr. Bucky Schwarz's Javascript presentation back in 2011(?).
\nSometimes conferences can be a bit stuffy and maybe pretentious. Not here, no way.
\nOf course the sessions are top notch as well - well worth probably 3 times the $200 ticket price. The only problem I had before with the sessions of past was that there was a lot of Adobe Coldfusion ones. Obviously not a problem for a Coldfusion developer, but I am not one. So my session choices were a bit limited during the first years.
\nLast year, it got way better. NCDevCon beefed up the web, mobile, and tools track. There was still plenty of CF to go around. This year - I see a Coldfusion, Javascript, Web/HTML5/CSS, and Tools/Technique track. In the same spirit as last year - but I did notice something. The Coldfusion sessions don't LOOK like Coldfusion sessions. I dare say they might even be language agnostic if the session descriptions are any indication.
\nThis means that someone like me actually might stand to learn a thing or two by wandering in to some of the CF sessions and look at the techniques for an interesting and foreign take to bring in to my favorite server side platform - whether it be Node, PHP, Python, Ruby, or whatever.
\nBut who knows, maybe you're one of those sheltered people who think that no good can ever come of Coldfusion and people are dumb for using it. Yes you exist, I hear you whine. That's annoying, but certainly doesn't preclude you from attending the other awesome sessions that don't mention the C-word.
\nBut me? I will have just moved to the bay area and hopefully doing the closing on our North Carolina house . I'll have also JUST started a new job. No time for NCDevCon love, Dr. Jones. That's why they suck.
\nIn fact I'll tell you about some specific sessions that suck - only because I won't be there to witness them:
\n\nHANDS ON WITH THE NEW KID ON THE JAVASCRIPT MVC FRAMEWORK BLOCK: EMBERJS by Ryan Anklam
\nRyan you suck. I wanted an intro to Ember without actually having to dig through docs.
\n\nJAVASCRIPT THE STACK by Adrian Pomilio
\nAdrian - you suck. You always give a great preso, and I'm learning all this stuff right now: Grunt, Less, Node, NPM, etc
\n\nSTOP MAKING EXCUSES AND START TESTING YOUR JAVASCRIPT! by Ryan Anklam
\nRyan - here you go again. Sucking times 2. I'd love to see approach on this.
\n\nA BRIEF INTRODUCTION TO CRYPTOGRAPHY FOR DEVELOPERS by Jason Dean
\nJason - you suck, but you might get a pass. I'm not sure if I need this, but it sounds way cool.
\n\nOFFLINE-FIRST: AN APPROACH TO MODERN WEBAPPS by Calvin Spealman
\nYep, Calvin sucks too. Would love to see this.
\n\nAs you can see - I'm very bitter and would love to go. Do not pass this conference up!
\n\nhttp://www.eventbrite.com/event/6398007613?ref=ebtnebregn
\n", "date_published": "2013-07-23T00:00:00Z" } , { "id": "https://benfarrell.com/blog/2013-07-22-ideas-from-flex-visual-application-states-in-angularjs/", "url": "https://benfarrell.com/blog/2013-07-22-ideas-from-flex-visual-application-states-in-angularjs/", "title": "Ideas from Flex: Visual Application States in AngularJS", "content_html": "Macrodopache Flex. That's what I'll call it - because during it's different incarnations it was managed by Macromedia, then Adobe, and now Apache. This brief intro is for all the Flash haters out there.
\nFlex is a "Rich Internet Application" framework built on top of Flash. That's fancy talk saying it was Flash for banks and enterprises and big intranets. I got into Flex at version 3 - at the point where it was reasonably priced and not server side.
\nI didn't work for a bank, I got into it because, as a Flash developer, I was sick and tired of creating buttons and dropdowns and other common UI components from scratch. Flex had a way for me to sweep the boring repetitive bits under the rug and work on my unique snowflake of an application/game/interactive.
\nWhat was frustrating about Flex 3.0 was it's ability to be skinned at any sort of deep level using any asset you wanted. I went deep down the rabbit hole in making Flex compiled libraries out of Flash to import them and use the assets. Overall, it wasn't pleasant.
\nThen came Flex 4.0. Lots of folks complained how different it was, but for me it was awesome. Most of the awesomeness was the ability to use "states" and design add any custom elements - vector shapes, bitmap graphics, or whatever to your states.
\nThe simplest example of this was the button. A button has an "up", "over", and "down" state. "Up" being the normal look of the button, "over" is what it looks like when the mouse is hovering over it, and "down" is the state when the user is in the process of clicking it. You can add more states of course - like selected for a toggle style button. The main point of this is to say that there are visual states.
\nIn reality - what is happening here is that all of these states exist, and a "state" variable dictates which of the 3 visual layouts/designs to show. It didn't really care what the contents of your visual state was - you simply defined the look of each state and what kind of component it was, and Flex would handle the rest.
\nSometimes your component might have some bound piece of information inside. Like a labeled button: you might create a button labeled "click me". This label needed to be identified in a common way and bound to the component available in all states you wanted the text to show.
\nBigger than that even - was the concept of "Application state". Really - take the concept of a button, and scale it up, defining whatever states you want.
\nSay I have an application that allows a user to log in, list people, select a person to edit, and then log out. My states might be "login", "list", "edit", and "logout". In HTML/JS/CSS world, we might have those on separate pages? How about a single page app?
\nYes, there are numerous ways to do it - but how about states? Take this concept in Flex (and yes it's common to any number of platforms - I believe I've seen it in MS Silverlight and Android) and adapt to Angular.
\nFirst - how to mange if one visual state is shown or hidden? Well - how about creating
<div> class="state login_container">....\n<div> class="state list_container">....\n<div> class="state edit_container">....\n<div> class="state logout_container">....\n\nAnd then in our CSS - the default style for each state would be:
\ndiv.State {\n display:none;\n}\n\nThere! Now all states are hidden by setting the display to none. They aren't even in the DOM's render tree affecting the layout - fantastic! This means, I can show one or the other, and they will appear in place - right where they should be, not affected by the others.
\nHow to show them now? Well, first we need to have some sort of "state" variable to indicate which state we should currently show. Since we're in Angular, we can put this in our controller to enable binding:
\nfunction AppController($scope) {\n $scope.state = "login";\n....\n\nNow that we have something to bind to - lets do that in or Angular markup/HTML:
\n<div> class="state login_container ">....\n<div> class="state list_container ">....\n<div> class="state edit_container ">....\n<div> class="state logout_container ">....\n\nAnd then, pop over to CSS:
\ndiv.State {\n display:none;\n}\n\ndiv.state.login_container.login,\ndiv.state.list_container.list,\ndiv.state.edit_container.edit,\ndiv.state.logout_container.logout {\n display:block;\n}\n\nSo as you see - we include each state we want in the selector that says we should display as block. We select those elements that are div.state.someState_container, and only use this style when the proper state is paired up with the proper container in the div tag.
\nFor example - div.state.login_container will select login regardless of the state, but with the state variable bound to the class - div.state.login_container can be div.state.login_container.login or div.state.login_container.edit or div.state.login_container.list, etc.
\nOnly when it is bound with Angular and can be selected by div.state.login_container.login do we set the display to block.
\nIn this regard, we are showing and hiding application states based on Angular binding!
\nLike my last post, there's really nothing magical - it's pretty simple, but a potentially nice way to manage visual elements if it is right for your project.
\nA word of warning, though - the contents of your states, while removed from the DOM should still be active in memory. Worse - I'll bet that any Angular bindings on your inner content will remain firing and active. So take heed to not pack your states too full of things. You can't see them, but they are there and working.
\nPersonally, I've only done this for application states, but I do wonder if one could create some nice components here based on this (or maybe if someone already did!)
\nThis is actually a pretty simple concept - but it might not be intuitive for everyday web applications. I didn't even think of doing this until I started making a game (of sorts) in AngularJS.
\nThe concept is a "resource library". I've come across a few different game engines in Flash and elsewhere, and they seem to have this design pattern in common.
\nThe raison d'etre of the library is as follows:
\n\n\n\nYou have a game. In that game you have a decent number assets: sprites, character animations, sounds, etc. You'll be dealing with a situation that you want to be certain that all assets that you need to play the game when you load it are actually available. Rather than load each thing individually in code, we create a structure called a library. You can set resources in that library, and get them later by name (or by any complicated syntax of your choosing).
\n
Now, compare this to a method that jQuery users know - caching an element so that it doesn't cost much to reference it repeatedly when we need it. Sorta similar, no? Add to that a nice little convenience method and structure for getting, setting, storage and retrieval.
\nHow about a little flare on top of that?! Use an AngularJS directive!
\nHere's a simple Library object:
\nlibrary = {};\nlibrary.elements = {};\n\nlibrary.setResource = function(name, elem) {\n library.elements[name] = elem;\n}\n\nlibrary.getResource = function(name) {\n return library.elements[name]\n}\nNow, as you can see - we have a couple convenience methods for setting and getting a resource. The elements get popped right into the library.elements hash table - pretty simple stuff.
\nTo get the element you want - just call library.getResource("myelement").
\nNow, originally, I had an initialization routine that ran through all of the elements I wanted by using DOM selection with the class chain, as you would do with jQuery or such things: $(".myelem")[0]. Note that the array index of [0] is to assume there is one and only one element found and break it out of that jQuery-like array structure and get the raw element we want.
\nIn my initialization routine though, I'd make some syntax errors and make a mistake targeting the actual element I want. It's fine - but can be a pain when things don't work, or maybe you do some HTML/CSS class restructuring and break everything.
\nFor example - maybe my resource was div.classA .classB. classC, but then I removed the outer layer that is .classA leaving .classB and .classC intact. Well, I just broke my library reference! Do I go back and edit - or maybe there's a more dynamic way?
\n\nAngular directives are awesome. To review what they do, in short, is that they allow you to make your own HTML attributes for existing tags (I believe they do process more than just attributes, but that's how I use them).
\nSo, I can write some HTML like so:
\nAnd write a directive:
\nMyApp.directive('log', function () {\n return function (scope, element, attrs) {\n console.log(attrs.log);\n }\n});\nWell, what do I have there? Just something simple that logs a text message to the console, but does it from a freakin div tag! How cool is that?
\nMaybe you already know how directives work, that's cool....lets move on to our resource library. Lets make a directive that processes the following and adds the element to our library:
\n<img src="assets/player.png" library="player" />
So, in the above HTML, I have a simple little image tag showing a graphic. Later on, I'd like to reference it and do some CSS transforms on it. How can I reference it? Just by calling library.getResource("player")! This will return the element, already cached, so no traversing the DOM to look it up.
\nBy adding that library="player" markup, I've created a custom attribute that adds THAT specific element to my library and allows you to retrieve it by the name "player".
\nOf course, it is custom - so we need to write the Angular directive ourselves:
\nApp.directive('library', function () {\n return function (scope, element, attrs) {\n library.setResource([attrs.library], element[0]);\n }\n});\nSo in our directive, we're simply taking the element (first in the array to get the raw element, assuming it's unique), and setting that resource in our library by whatever name our "library" attribute was given.
\n\nI have no need to get fancier yet. My application/game/whatever is pretty small and there are only a limited number of resources I need. And, I want them all available when I load it up.
\nHowever, what if you had 10's or 100's of elements? Memory concerns aside - you might have some difficulty giving them all unique names and might want to introduce something like folders.
\nI think I recall a game engine using something like a dot syntax for resources -
\nDoing something like library.setResource("folder", "resourcename", element) or library.setResource("folder.resourcename", element) will create a structure like so:
\nlibrary.elements["folder"]["resourcename"] = element;
\nIt can really be as multidimensional and organized as you want to be.
\nAlso - perhaps you'd like to attach elements dynamically, load resources on the fly at different points in your application. Additional logic can be added to your library object for such eventualities.
\n\nAnyway, the resource library is a pretty simple concept, but pretty handy! Not something I thought about introducing until I was in a game mindset.
\n", "date_published": "2013-07-15T00:00:00Z" } , { "id": "https://benfarrell.com/blog/2013-06-03-back-from-jsconf/", "url": "https://benfarrell.com/blog/2013-06-03-back-from-jsconf/", "title": "Back from JSConf", "content_html": "Oh good, someone already posted the "dripping with love" review of JSConf and organizers Chris and Laura Williams:
\nhttp://adambrault.com/post/52004360603/just-a-person
\nI'm so glad that's out of the way because I really didn't want to do it. It's not that JSConf wasn't crazy good (it was), and it wasn't that Chris and Laura didn't go completely over the top in a good way for everyone involved (they did), it's just that I don't want to be the guy writing those posts because there's so much more good to talk about - especially when I'm pretty certain if you asked ANYONE at JSConf if they'd recommend going next time you'd get a yes. Seriously, anyone...I really can't think of a thing wrong with the conference aside from the horrible cellular coverage in Northern Florida on T-Mobile while I'm selling a house and looking for a job.
\nSo with that out of the way, dripping love for JSConf aside...
\nThere was an undertone to this conference and the community there. Maybe I'm reading into it because I want it to exist - but I don't think that's it. I think it was really there: in the sessions, in the people, all around.
\nJavascript has a community both young and old because of the nature of the newfound revival of a language that's been used for 18 years. Javascript is also a language that nobody can tell you the "right" way to do things. There is no "one" MVC framework - in fact, there are oodles.
\nGiven how often we adopt, drop, and rework our flow in Javascript, when large groups of us meet to talk about Javascript it's chaos. And I love the chaos - it's like a mix of hardcore language basics and boundless creativity and rocketscience from all walks of life.
\nLet me explain with the rise and peak of Adobe/Macromedia Flex that I lived through:
\nI used to use Flash quite a bit - it was my favorite platform several years back (though I didn't have that much experience in others at the time). I was drawn into it by the creativity being sparked in the community. It was THE creative tool to use, hands down. Something happened, though - we adopted a framework. Flex is/was pretty damn good actually. It shares a lot with other similar things like Silverlight/XAML and Android/XML. Nothing wrong with Flex.
\nHere's the problem, though - I remember being at a 360Flex (great conference BTW) several years ago and EVERYONE was doing business applications and throwing Flex at it. There really wasn't much talk about other tools/technology/creativity - just how to do stuff in Flex. I know, I know - Flex conference...but it really highlighted the shift in my favorite creative platform to the business side of things. Everyone was talking about Flex, not so much about Flash - just Flex. Adobe pushed that stuff hard to Java devs, and with the huge marketing focus on "Flex: The One Framework to Rule Them All (better than crappy Java) ™", Flash was sort of forgotten about by lots of money seeking entrepreneurs and also folks who just didn't want to get left out of the new technology standards.
\nAt the time, I thought that the entire community settling on a single framework was a good thing - but for me, it really sucked the creative juices and talent from an awesome animation/interactive platform.
\nCompare this to what I'm witnessing in the Javascript community. "OMG I love Dojo...wait no! Backbone is for me! But look over there, it's Google Clo....oh wait its Angular now!" We're talking about MVC frameworks, and we're talking about boring old business applications - but things are moving too swiftly to settle in. As a result, we can have rigorous and intelligent conversations about Require.js, but we're not going to get it crammed down our throats at JSConf.
\nThe lack of a solid direction here is a huge opportunity that I'm seeing folks at JSConf take in spades. Instead of obsessing over one thing -we're obsessing over a huge number of things in completely separate directions. This is what makes a community absolutely fun and thriving.
\nWe had Janessa Det obsess over beat detection and sound. We had Steven Wittens obsess over WebGL, we obsessed over Nodebots, Nodecopters, and 3D printing. Would we have seen this if 2 years ago someone had declared Backbone.js "The One Framework to Rule Them All (ps: Flash sux)™"? I'm obviously not a JSConf organizer, and having met Chris Williams once for 4 seconds, I certainly can't speak for him. However, perhaps I'd think as an organizer, I should load my conference up with Backbone tracks and Backbone training.
\nAs a result of all of this, I'm seeing loads of creativity and creative uses of Javascript - much of this knowledge coming from communities and research that has nothing to do with Javascript or the web. This makes me very happy.
\nPerfectly capping it all off, we were treated to Seb Lee-Delisle. I truly hate being a fan of anyone, but its really hard not to be a fan of Seb's. In addition to showcasing some jaw-dropping interactive installations, he treated us to a short Creative.js training session with generative art in the HTML canvas. His session was the perfect mix of humor, self-deprecation, humbleness, and inspiration. He had one piece of great advice that I've only been smart enough to adopt recently - "Think small and FINISH YOUR PROJECT - Nobody will see it or know about it if it's 90% done".
\nSeb's was the perfect end session to spark a thought in my head. The thought is that this community has some very creative and curious folks. And for me, knowing that there are lots of other creative and curious folks creates quite the inspiration to continue what I'm doing....and of course finish more of what I start!
\n", "date_published": "2013-06-03T00:00:00Z" } , { "id": "https://benfarrell.com/blog/2013-05-28-adventures-in-silicon-valley-job-hunting/", "url": "https://benfarrell.com/blog/2013-05-28-adventures-in-silicon-valley-job-hunting/", "title": "Adventures in Silicon Valley Job Hunting", "content_html": "Admittedly, I've been quiet lately...too quiet. That's because I'm taking the plunge and looking for work in the California Bay Area!
\nFor me, a UI developer, the North Carolina Triangle Area, while a hotbed of awesome tech and startups, is still too data-centric. I don't mind data -- data is good, but I want great interaction and visualization to back it up. To be fair, I did tell someone my line of reasoning and they offered to hook me up with several places that offered that if I'd stay in town.
\nBut our minds (my wife's and mine) are made up! I think I'm making the right decision. Looking on Meetup.com, I see 20 times the interesting meetups in the Bay Area than in the Triangle. Browsing Craigslist, I see 2 RTP area jobs per day in the internet engineering section, whereas in the Bay, I see 20 per day.
\nThese are the things I justify the decision with, as right or as wrong as they may be.
\nSo as a result, we are selling our house, moving (probably) to the Bay Area, and I'm finding a new job. As anyone in this nerve-racking situation would do, I jumped in immediately to search for jobs from afar -- and that's how I've come to learn the ways of Silicon Valley style job hunting. I think I finally have a decent feel for it, though I've done a lot of stumbling.
\nI'm of the opinion that my resume is fairly strong, so getting callbacks hasn't been a huge issue -- but they do take a while. Most have taken around 2 weeks for a response from the time of submitting online. Submitting online is a huge hassle but one you just put up with. You're subscribing to any number of recruitment sites to submit your resume and cover letter. Brassring.com comes to mind. Or if the company is big enough, they have their own internal submission tools. Either way you're entering in your information over and over and over again. Even when linking your profile with LinkedIn, you generally have to enter it somewhere else for each job.
\nAnnoying, yes. End of the world, no.
\n\nNext you talk with a recruiter. I've actually enjoyed talking with the recruiters -- you get to brag a little, and it's their job to stroke your ego and reel you in. The bigger companies have whole recruiting departments. In one case, I was passed through 4 different people before I landed on the one that could help me out.
\nI think the only way you'll screw things up at this stage of the game is to start swearing profusely or making death threats. Of course, I've passed with flying colors -- making 0 death threats. I haven't even WANTED to drop the f-bomb; they have simply been pleasant people to talk with. When you pass this phase, it becomes their job to schedule a tech interview.
\nPrior to my first ever failed Silicon Valley style tech interview, which was with Google, I didn't know what I was getting into. The recruiter basically said, "You know the drill." I thought to myself, "Sure, I know the drill. We'll get technical and maybe throw in some questions to prove my domain knowledge." Nope, I was unprepared for this next world of hurt.
\n\nSo, I had an appointment at a specific time. I was also passed a Google Doc link so the interviewer and I could share a screen and collaborate via code. That was new to me, but I rolled with it, unsure of what would happen.
\nSince I WAS unsure, I reached out to one of my wife's friends who worked at Google to learn what to expect. He told me to read the book Programming Interviews Exposed.
\nI picked it up on my Kindle -- but I was already at t-minus 2 days until my tech screen, so I skimmed. While I skimmed, I recognized that it was a pretty decent book, but I didn't think it was AT ALL applicable to my situation. After all, I was a UI programmer going for a Javascript role for god's sake. I thought that as long as I could express some domain knowledge and maybe explain what my jQuery was doing under the hood, I'd be set.
\nThe book covered sorting, trees, binary trees, graphs, implementing linked lists, Big-O analysis, and more. I must say that I brushed it all off before my first interview. Those are concepts I really haven't discussed or seen since my college courses (13 years ago), and especially given the discussion on pointers in the linked lists and graphs, I really thought I'd be more preparing for a C++ interview with this book....not a Javascript/HTML one!
\nI was wrong, so wrong. Try giving a friend or loved one a call RIGHT NOW and explain how to do a breadth-first search on the phone without first refreshing your memory. It didn't work for me. BIG FAIL for me in fact. In fact, the breadth-first search problem is a great example of what to expect on a couple different levels.
\nFor me, being asked that proved that not all code you'll be writing in the interview has a sexy, elegant solution when you're on the phone with your interviewer. For those that don't know breadth-first search, you're basically taking a tree like an XML file or any other nested structure. Instead of searching the root, then the child, then the child of the child, you're searching the root, then all the 2nd level, then all the 3rd level. The solution here is to push all the child nodes into a stack and shift off the first one with the important thing being that you need a list/array to store things in memory.
\nI didn't even want to go there for that question and other questions in the interview. I thought there had to be some easy, recursive solution that spanned like 3 lines of code. Why would they ask something overly complicated and unelegant over the phone?
\nWell, they do. They want to see error checking. They want to see the solution as messy as it may be.
\nIn the end, that book proved to be the perfect preparation guide, which I was able to absorb for my subsequent interviews.
\nOne other thing this book teaches is that it's OK to stumble if you don't know the solution to a problem that the interviewer asks. It's necessary to just keep talking to them. Don't choke, just keep talking. Tell them why you are going down a path to solve the problem and what you are thinking on the fly. If you don't solve it, they may give you some hints. Even if you think you fail at solving the problems, they may be impressed with your problem-solving process and still continue on to the next round of interviews.
\nMy first interview with Google? I did NONE of that. And you know what else? I never got called back. After that bomb of an interview, I was able to have some really good ones based on what I learned.
\nAfter all that, I've had some good interviews -- but my best ones have been when the interviewer asked for domain knowledge. You know, that interview where it's not like an interview, but you just nerd out and talk about what you do at your job and what you like or dislike. Those are the ones that have made me feel like a real person, not a piece of meat, and it's shown.
\nEven my Google interview might not have been my worst. That was probably the one when the interviewer was a guy less than 1 year out of college. I don't say his age to be dismissive. I have no doubt he is very smart, but he didn't have much experience giving a tech interview -- which meant he picked random problems out of a data structures and algorithms book. But I kept talking and giving him my thought process, despite getting every question potentially wrong. To my surprise, I got a callback for a second interview after that debacle, which was more of a tech conversation that went quite well.
\n\nRight now, I have a few on-site interviews lined up. The kind where you get flown out like royalty and put up in a hotel room. My initial reaction to getting asked out for them was that they must really be formalities, and if a company drops that kind of money on me, it's mostly a sure bet I'll need to screw up pretty bad to get rejected. But given how wrong I was before, I'm not taking any chances! Prepare for the worst is what I'll do!
\n\nAs a final thought, I want to talk about how weird it is going for a front-end job these days that focuses on HTML/CSS/Javascript. Even the domain knowledge interview is a disaster. There is little chance you'll talk about the same frameworks in most of the interviews given there are so many popular ones (especially MVC style). To top that off, you really do feel like an idiot if they ask native Javascript questions. No, I am not a jQuery scripter that copies and pastes. However, because I use jQuery a lot, I sometimes forget the exact native Javascript syntax . See the problem? To prove you're a bad-ass Javascripter, you must recite things you never use in your day-to-day life.
\nI also got a rather nice interviewer once who unfortunately didn't really know a thing about Javascript/UI. I ended up switching gears to talk about Java and back-end stuff. Front-enders are in demand, and as a result, you might be interviewed by someone sorely needing a front-ender...to the point where they have none to interview you properly!
\n", "date_published": "2013-05-28T00:00:00Z" } , { "id": "https://benfarrell.com/blog/2013-05-07-my-first-node-js-plugin-for-3d-motion-sensing/", "url": "https://benfarrell.com/blog/2013-05-07-my-first-node-js-plugin-for-3d-motion-sensing/", "title": "My first Node.js plugin for 3D motion sensing", "content_html": "Well that's a little bit of an awkward title - its not ONLY my first Node.js plugin I've released on NPM but it ALSO does 3D motion sensing! So brand new on both counts - I'm no Node.js plugin/addon veteran by any means!
\nYou can grab my NuiMotion project version 0.1 on NPM and Github. You can read why I did it, and about how I'm on a crusade for letting your interfaces move your body on the project page.
\nThat said, I learned a lot of stuff. I think the project serves as a shining example of how one guy accomplished some rather difficult and not so ordinary things you'd need to do with Node.js. I won't claim it's necessarily the right way - just one way. I was a little scared of C++ before all of this, but I jumped in because I had a need that I wanted to fill, and C++ was the only way to get it done.
\nThe C++/Javascript bridge is pretty cool to see how it works. There are all sorts of problems you run into with type conversion - your C++ variables to those loose Javascript variables.
\nThe hugest hurdle was breaking down the architecture to something that wouldn't block the entire Node.js main process. In all of the OpenNI examples, they would run a big old while loop that will grab frames of video/depth and pull out the features we need like gestures and skeleton data.
\nThis is SO not cool for Node.js, so I needed to delve into how to achieve threading in the V8 engine with "lib_uv". I still don't understand everything about the final lib_uv code I used (why some things are declared as they are), but I successfully broke it out into a new thread that runs as fast as your machine will let it. We reach in and grab our joints using a custom defined interval to poll at, and we event out when gestures and other events are encountered.
\nOf course, all of this NEEDS to be threadsafe. If you access the wrong thing inside a thread, you crash your entire process.
\nYou can checkout the main logic of all of this, completely with C++/JS communication and threading here:
\nhttps://github.com/bengfarrell/nuimotion/blob/master/src/Main.cpp
\nI didn't do this alone, either. I asked a couple questions to the extremely awesome Node.js Google Groups. One was around the threads question, and the other was around C++ compiling. To demonstrate how much of a noob I was, my compiling question was that I didn't realize you had to include ALL your *.cpp files in your sources target. I thought since the main.cpp references other things, they would be automatically included. NOPE! Live and learn.
\nAnyway - I'm of the opinion that this project probably represents some of the most difficult things you could ever need to know how to do in a Node.js addon (without getting into domain specific C++ code which could be infinitely complex for sure). So feel free to have a gander and learn!
\n", "date_published": "2013-05-07T00:00:00Z" } , { "id": "https://benfarrell.com/blog/2013-04-19-i-got-my-oculus-rift-vr-headset-today/", "url": "https://benfarrell.com/blog/2013-04-19-i-got-my-oculus-rift-vr-headset-today/", "title": "I got my Oculus Rift VR Headset Today", "content_html": "Ooooog. I've been noodling around with this thing for about an hour. I think I'm at a good stopping point because I'm a little motion sick. It's weird because I think I'm only getting motion sick when I take OFF the goggles. Maybe I just can't adapt to real-life anymore.
\nSo, for those that know what the Rift is...awesome. But those that don't - it seems like it may be the first legit, consumer-grade (read: affordable) VR headset.
\nThe Oculus Rift is the very first thing I've ever backed on Kickstarter (my wife has since backed the Veronica Mars movie). But the experience was harrowing. I mean - you're waiting for this thing for almost a year after dropping $300. At the time, I thought $300 was a pittance to pay for something so potentially cool.
\nThen the delays started happening. Oculus was supposed to have this thing out in my hands for December. I just got it now in April. It was supposed have Doom 3 support when I got it - it doesn't, ID Software backed out I guess. Bad news? No, actually....just no. Oculus, the company behind this, with these setbacks COULD have been so shady about all of this, but instead they've been an extreme class act every step of the way.
\nSomething this cool was bound to have setbacks, and yah...Kickstarter...I'm probably lucky I got anything for my $300. Oculus has been very communicative every step of the way since August. They've done interviews, showed their tech off at game conventions, everything. They kept me drooling and happy to wait with them through the problems.
\nNo Doom 3 support? Man that sucks. But, the second this was announced, Oculus stepped up and offered a complete refund, a Steam Store credit, or an Oculus store credit for $25. Way to go, guys - even after Team Fortress 2 support was announced in lieu of Doom. Yah, there were setbacks, but they handled it like champs.
\nImagine my....OK...non-surprise (since I had the shipping notification this week) when I got home and my Oculus was there waiting for me. Given the trivial amount (yes I think $300 is trivial for a VR headset) you'd expect them to do a shoddy job packaging it. Nope. I have a hard case with foam cutout inside that molds itself to all my pieces. What the hell guys....this is more pro than real store-bought finished products. I was even more impressed that it came with DVI and HDMI cords to connect my machine to the breakout box. They could have made me go to the store to buy these, but nope - comes with.
\n![\"13](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2013/04/13-1.jpg\")
![\"13](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2013/04/13-1-1.jpg\")
Whatever though....what's the experience like?
\nIt's incredibly fascinating. The best demo so far that I've seen after playing around for an hour, is the "Tuscany" demo. This seems to be their showcase piece for now. Basically, they've created a Unity demo with a quiet little cottage in Tuscany. You start out in the cottage in front of a roaring fire. Putting on your goggles for the first time is a very wowing experience. It's so unusual, but your brain adapts pretty quickly - so quickly in fact that when I looked down and put out my hands, it was very disorienting to not see my body when I KNEW it was there.
\nYou look around and you feel like you're there. The roaring fire is a nice touch. It was a little awkward fumbling around to feel where my laptop keyboard and trackpad were, but I eventually found the keyboard controls and walked outside (no bounce to my step....that woulda been way cool).
\nAfter wandering outside, I immediately stepped towards the ocean and had a sensation that I didn't want to get too close or I'd fall down the cliff. After getting over this, I looked around - dandelion particles were drifting through the air...a VERY nice touch.
\n![\"tuscany_smaller\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2013/04/tuscany_smaller.jpg\")
The extreme novelty wore off after a bit of wandering around, which was then around the point where I started to scrutinize things. Walking up to things and looking close of course revealed the flat/3D polygonal look of the trees and other objects that from far away look awesome. You can't NOT expect this really - I mean, realtime 3D environments are what they are, but it was interesting to experience this!
\nThen if you shift your focus, you can make out the individual pixels on the screen - as if you were inches away from a TV. You can easily choose to forget this and enjoy the experience, but it detracts...just a little bit. Now that I think of it, one of the little demos that I downloaded from their forums by a fellow developer was mostly over a black sky, and generally dark, which removed the razor thin black border that goes along with these pixels.
\nThe biggest detractor was the head movement. The best I can describe it when you move your head fast is choppy. It felt a little like the headset was polling your head's movement at a lesser interval than needed to stay smooth - and the view in front of you chopped along. My wife actually made the stellar point that using the mouse to move didn't produce this - so my theory of refresh rate was shot down. Also if you take off the headset and move it with your hands while you watch your 2D computer display, it's not noticeable on there. There must be some weird headset/display refresh rate combo effect going on. Also, slower head movements didn't do this for me. Another demo seemed worse and another better - so I think this, my biggest issue so far, may just be a software improvement - and this is A-OK with me!
\nAnyway - I dreamt that this thing would be perfect. It's not, but it's more amazing than the cynic in me expected. There will be improvements as we go along - Oculus NEVER said these dev kits would be consumer grade yet. I'm looking forward to playing more and definitely checking out Team Fortress 2 this weekend!
\n", "date_published": "2013-04-19T00:00:00Z" } , { "id": "https://benfarrell.com/blog/2013-04-03-painting-with-chromes-webspeech-api/", "url": "https://benfarrell.com/blog/2013-04-03-painting-with-chromes-webspeech-api/", "title": "Painting with Chrome's WebSpeech API", "content_html": "A while back, when Chrome 25 was still unreleased, I tried out the beta version. I was directing my attention at the "WebSpeech API". It failed MISERABLY. I don't know if the ambient noise in my room was having a bad day or what, but I spoofed their "Writing your Dad a Message" video that Google put out.
\n/blog/2013/01/31/i-wrote-my-dad-a-letter-with-the-webspeech-api/
\nBut that's all in the past - I'm blown away with how it works now. True - if you have a good deal of ambient noise in your room, it looks like Chrome has a hard time differentiating between you and the background noise. Given a relatively quiet environment, or if you're up close to your microphone, it works magnificently.
\nI liked it so much, I created a little painting demo with it and a Kinect like depth camera. You can still use your mouse to paint in my demo though - so even if you don't have a Kinect working over websockets or the like, it's still functional!
\nI set up several keywords that the WebSpeech API listens for: red, green, yellow, purple, blue, orange, and black. These voice commands are to switch colors. I also added a "start over" command to clear the canvas. Lastly...invoke "Bob Ross" for a little surprise!
\nChrome listens to your speech, chops it up, and sends requests over to its servers. You'll get back "interim" speech and "final speech". Interim speech doesn't make too much sense as you see it. It's putting together it's best guess at phrasing as you go along, picking out the words it can without paying much intention to intended meaning. The "final" speech comes delayed by a second or two. Chrome things you're finished speaking, and it waits a tick to make sure. Then it puts together it's thoughts and gives you something final. It's probably going to be grammatically correct and nicely accurate.
\nSome of the shorter commands you'll need to enunciate. It has a big problem when I say "green" sometimes, but honestly, when I'm making voice commands, there's really no context in my grammar to pick up on.
\nI went ahead and wrapped up the WebSpeech API for Chrome 25 in a little Javascript library. I allow you to add commands to listen for along with callback methods to call when those commands are spoken. I also allow it to "keepAlive", which means if Chrome thinks you've stopped talking for good, I go ahead and start the speech API right back up. I also event out the surrounding words with a command if you care to make something like "search" a command and want the words that we spoken after it.
\nYou can checkout the library I made here (its in a repo I made for motion/gesture control as a demo to add speech to the mix):
\n\n\nOne annoying, but understandable thing is that to allow the Web Speech API to work, the user has to click to allow the microphone. This needs to occur every time you start the API. There could have been an error, or it could simply think you're done talking - then when you go to start it up again, the user needs to click to allow. Bad for natural interaction if you'd like to keep it running and listening at all times on your page. GOOD if you don't want web pages spying on you!
\nYou can work around this by serving your page over HTTPS. If you're secure, then Chrome allows you to always remember the option if the user so chooses.
\nIt's a very cool feature that I hope the rest of the browsers adopt soon! I wouldn't mind dropping the HTTPS requirement though in favor of something else!
\nFor my horrible Bob Ross impression AND demo, check out the video I made here!
\n", "date_published": "2013-04-03T00:00:00Z" } , { "id": "https://benfarrell.com/blog/2013-03-30-drag-and-drop-with-angularjs-and-jquery-ui/", "url": "https://benfarrell.com/blog/2013-03-30-drag-and-drop-with-angularjs-and-jquery-ui/", "title": "Drag and Drop with AngularJS and jQuery UI", "content_html": "I actually haven't used jQuery UI much, and been just tooling around with AngularJS. That's why when a recent project came up at work which had the typical ASAP schedule, I thought: "Let's put AngularJS to work!" Also, the application I was building was an internal tool, so it could be a little rough around the edges and customers/clients wouldn't make fun of me if something wasn't quite right.
\nSo....low risk, let's dive right into Angular! I also had the assumption that if Angular was all I thought it was cracked up to be, it would save me lots of time, and allow me and others very nice visibility to how my project worked so they could make updates very simply, right in the HTML.
\nBoy was I right. I felt like Angular sped things up and gave me what I wanted, with a nice easy, predictable structure.
\nWhat I didn't expect was that I'd be mixing jQuery UI in. What I REALLY didn't expect was how SLICK Angular makes it to mix things like jQuery UI in.
\n\nPretty simple right? Of course, of course I can write this from scratch in Javascript. And if I wrote it from scratch, I'd probably cut corners and.....not necessarily make it all fancy with tags and attributes for my Angular setup. No, I'd probably loop through all the elements in my list that I'd like to drag, attach onmouseout, onmouseover, and onmousemove listeners. And then keep a separate list of all my drop zones and check positions of things while they are being dragged.
\nPAIN IN THE BUTT. I don't mind doing cool things by hand, but things that are readily available elsewhere? Boring. So I turn to Google and type in "AngularJS drag and drop". I came across this gist:
\nhttps://gist.github.com/codef0rmer/3975207
\nGiven what AngularJS is supposed to do, I didn't expect to find any functionality in Angular to support drag and drop. Indeed I didn't, but what this gist shows is the ability to declare your own attributes in HTML and have Angular process them, adding your own logic to what those things do. THAT IS SLICK!
\n\nSo let's take "draggable" and "droppable" as our primary examples for an Angular directive.
\nIn our HTML, create a div with your own custom attribute on it:
\n\n
\n
\n
\n
Obviously these two attributes mean nothing to HTML, so we want to process them with Angular. Luckily we've declared our Angular App as "App". We can then process our directives:
\nvar App = angular.module('App', []);\nApp.directive('draggable', function() {\n return {\n // A = attribute, E = Element, C = Class and M = HTML Comment\n restrict:'A',\n link: function(scope, element, attrs) {\n }\n}\nSo, above, we've made a reference out of our "App" module (our HTML page/app). Then we'll add a directive - which is simply saying, anything labeled "draggable" - return an object that does something.
\nI haven't tried anything except "restrict:'A'" yet, however this is pretty fancy. We filter this behavior by attributes, elements, classes, or comments. For our example, we're restricting by our attribute.
\nThen the link process. Here you define what you'd like to when Angular finds the attribute. We'd like to hook up the jQuery UI drag behavior:
\n element.draggable({\n revert:true\n });\nRight! So all we do here, is take the element that the Angular directive gives us and call the jQuery UI method "draggable" on it. I left the gist author's revert:true in there, cause it's pretty nice. If the drag isn't accepted, its animates back over to where it started.
\nSimilar deal with droppable:
\nApp.directive('droppable', function($compile) {\n return {\n restrict: 'A',\n link: function(scope,element,attrs){\n element.droppable({\n accept: ".itemHeader",\n hoverClass: "drop-hover",\n drop:function(event,ui) {}\n }\n}\nWe process the directive, filter by attribute, and on link call the "droppable" jQuery UI method.
\njQuery UI has some nice options here for droppable. It can be set to only accept items of a certain class or ID (comma delimited lists inside the quotes are cool here). You can specify a hoverClass as well. In my case, the "drop-hover" CSS adds a white border to the element. When you drag your draggable over a dropzone, then the dropzone will light up with a white border.
\nAnd then the drop functionality. What happens when you drop it? I have my own logic, you'll certainly have your own. An important note is how to get the drag or drop elements using these parameters that you get:
\ndragged = angular.element(ui.draggable).parent();\ndropped = angular.element(this);\nThis, of course is in the "droppable" method. If you were in the other "draggable" method, angular.element(this) would be the draggable.
\nWait theres more! I had to do a little trickery in my application. See, it turned out that not everything could be dragged to everything else. If you picked up a draggable, you can only drop it's related dropzones. Other draggables have other dropzones.
\nI needed a good way to NOT let the user drag something where they shouldn't depending on what they picked up, but also make them SEE what they can or can't drop on!
\n\nLuckily, JQuery UI gives us a few more draggable options:
\nApp.directive('draggable', function() {\n return {\n // A = attribute, E = Element, C = Class and M = HTML Comment\n restrict:'A',\n link: function(scope, element, attrs) {\n element.draggable({\n revert:true,\n start: function(event, ui) {}\n stop: function(event, ui) {}\nSo, we can have something happen when the user starts to drag something AND when they stop. So for me, when "start" is fired, I loop through my list of dropzones. If it's determined that the drag is not allowed on a given dropzone, I'll add a "reject" class to the dropzone. In my CSS, I fade the background and the text and everything so it looks dim. And then in my element.droppable.drop function, I check to see if the "reject" class is present - and don't do anything if so.
\nIn my stop function here, I simple go through and remove all the reject classes from everything, since the user has stopped dragging.
\n\nLots of options here for doing drag and drop! And a pretty nice way of Angular opening the door to other tools as we add non-Angular functionality.
\n", "date_published": "2013-03-30T00:00:00Z" } , { "id": "https://benfarrell.com/blog/2013-03-26-finally-picking-a-focus-sweatin-to-the-web/", "url": "https://benfarrell.com/blog/2013-03-26-finally-picking-a-focus-sweatin-to-the-web/", "title": "Finally Picking a Focus - Sweatin' to the Web", "content_html": "I jump around in technology like nobody's business. One minute you'll find me illustrating, another minute you'll find me doing web apps, and the next you'll find me doing games. And then I'll bust out some PHP or Python back-end
\nLast fall, I started playing around with the Kinect, and I was seriously in awe with how magic it seemed. I started reading up on NUI (natural user interaction) and went further down the rabbit hole.
\nMonths later, I'm still deep in it. I'm finishing up my Node.js natural motion add-on and demo content just in time for some upcoming speaking engagements. I'm still not bored. I'm still not bored, ready to move on - and my Leap Motion is shipping in May. My Oculus VR headset is shipping in a matter of days.
\nAlso after a few months, I'm still playing the augmented reality game "Ingress".
\nThese things all have something in common to me...and it's pretty much the Natural User Interaction angle. How do we interact with interactive experiences and apps in a natural way - whether with a GPS and physical location, or with your whole body as a controller.
\nAnother important thread here...I'm out of shape. I sit at the computer too much. Rather than make the above endeavors work against me, I'm going to make them work for me and not do anything in my spare time that doesn't encourage moving around a lot.
\nThis means if I develop an interactive experience, it needs to involve moving around. How much do you debug an app while creating it? A LOT. What if testing and debugging necessitated moving around wildly to do this?
\nSo here I am wanting to learn about existing technology that gets you moving and creating my own experiments that get you moving. Not only for the fun of it, but to get myself in shape! It is time for a non-technical hub for this, and I shall become a "Digital Fitness Trainer" in training. That hub is Sweatin' to the Web. There I will do video reviews of apps, sites, and gadgets that get you moving around. I'll also be posting videos of my creative experiments there, like my work with Kinect-like devices.
\nIt will be pretty non-technical there, but I'll follow up on this blog with the actual technical side of things - how it works, maybe dig into the code if appropriate.
\nI'm excited! I have a Fitbit review up now with a depth camera painting demo coming in the next week or so. Hope you enjoy, but if not...I'll keep blogging all the technical tidbits here as always
\n![\"sweatin-banner\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2011/12/sweatin-banner-1.jpg\")
I'm trying to avoid internet negativity at all costs lately. It's so passe. But I was having such a rockin time working with AngularJS today, I just had to tweet that it was my absolute flavor of the moment. Right after I did that, though, I had a reality check...will this turn into one of those boring old internet flame wars where my framework is better than your framework?
\nI got a reply right away. Luckily @jwaltonmedia was just being friendly. See, his flavor of the moment is BackboneJS. We traded some kind words and left it at that - each sharing some awesomeness.
\nGot me thinking though, there's too many articles out there about MY FRAMEWORK VS YOUR FRAMEWORK: THE ULTIMATE THROWDOWN. The winner rocks and we should push it on everyone whether they want it or not. The loser suffers in shame for eternity.
\nWell, I was inspired to compare a few of these frameworks, but I don't want to use negativity or competitive language. Because, in all honestly we are in a moment where there are a metric crapton of Javascript frameworks that are ALL GOOD.
\nThusly, I've named this article "fundown" instead of "throwdown", because the only winner here is you when you try all this good software. And instead of "versus", we're going to use "plus", because while it's not necessarily ideal to use 2 or more of these frameworks together, you should use all and more separately as you rock out your badass web app.
\nI've tried just around 3 frameworks/libraries heavily at one point or another - and I truly believe that each is the best tool for some job. My learning process was pretty unscientific and involved me copying and pasting tutorials and code from the web - so my opinion here will be equally unscientific.
\nIt's pretty easy to see that the 3 I've looked at are pretty mature at this point - Backbone.js JUST reached a 1.0 milestone and been around since October 2010. Angular is ALSO at 1.0+ and dates back to January 2010. Dojo is the grandaddy of them all, REACHING 1.0 in 2008. So, I think it's safe to say all mature, and all are well loved by their communities.
\nAll 3 are geared towards helping you get going to mainly build web applications vs web pages. Many times this will be single page apps, other times there will be lots of event driven data loading and interaction on the page....very AJAX-ey, or whatever it is the cool kids are calling it these days (especially since the cool kids are using JSON and not XML).
\n\nSo, by virtue of just being Javascript, if you screw up something (and you will because you're learning), it will silently fail on you. All of these are relatively complex systems someone set up. You have to initialize and use it. Each come with their own dependencies, and somewhere along the line, you'll want to use one of those dependencies, and it will fail on you. It's up to you to figure out how you screwed up the initialization of the framework/library. More often than not, you'll get something like "X doesn't exist", and you have to figure out what you did wrong in the setup such that X doesn't exist. Errors aren't the most helpful here.
\n\nDojo was probably the worst of the pack in terms of unknown setups screwing you further down the line. However, that's a testament to how damn powerful this package is. It probably packs the most power out of any JS toolkit I've used, and it comes at the price of complexity.
\nI would recommend not using Dojo for your one off web app. Use it for your enterprise! Many libraries are moving towards modular loading, but I think it's safe to say that Dojo had it first. And as well it should, because it comes with a crazy amount of components and tools. You definitely wouldn't want to load them all up front. I believe that most frameworks today will do modular loading, but they seem to make you realize that you are relying on Require.js as opposed to Dojo who advertises AMD, but doesn't really tell you it's Require.js - they seem to kind of brand it as a Dojo thing.
\nAnyway - Dojo....large amounts of skinnable modular components. Back when I was trying it out, they were just getting around to a proper MVC implementation.
\n\nSpeaking of MVC...holy hell, MVC is like Backbone's mantra. You're actually extending Backbone's Model, View, and Collection base classes. So everything, at least the beginner stuff, you see online is really going to push this way at you. And that's fine - but it's also why I've found that it was hard to get past the concept of a todo app really.
\nI'm not going to be stupid here and tell you that a todo app is the only thing you can do in Backbone. Backbone is plenty powerful and can do wonderful things - it's just my opinion that the beginner's material you see when researching seems to not really help you get beyond this simple "list my data" type of web app.
\nI didn't really care for the lack of HTML or otherwise markup used in the framework. It largely looks like you're encouraged to work with code and append to the DOM that way. Let's be honest though, on a large app, after you're settled in, doesn't it end up that way anyway? I mean, yah, it's nice starting out in HTML - but it starts to get lengthy and messy, and you just wish you modularized your stuff and programmatically did it at the end. One exception I've found here with Backbone is that people use a
", "date_published": "2013-03-22T00:00:00Z" } , { "id": "https://benfarrell.com/blog/2013-03-20-anatomy-of-a-kinect-like-gesture/", "url": "https://benfarrell.com/blog/2013-03-20-anatomy-of-a-kinect-like-gesture/", "title": "Anatomy of a Kinect Like Gesture", "content_html": "As anybody who's stopped by my blog over the last few months knows, I've been experimenting a lot with depth cameras like the Kinect and making it work as a plugin in Node.js. Happy to say that I'm moved the disorganized mess of experiments that I had in Github into what I hope would be a cohesive plugin.
\nI'd still like to test a bunch before I call it a release, but I think I have something going here to work a Kinect or Kinect like device through Node with the help of an open source library called OpenNI and not-so-open-source middleware called NiTE.
\nSending skeleton data, or joints, or body tracking points to Node was relatively straightfoward. I just needed to take the data and push it through, making it available to Javascript.
\nUnfortunately, though, one of the main things we associate with the Kinect isn't really included in the middleware I'm using, and you're left to your own devices. I'm talking about the gesture. Whether you're swiping your hand, waving, or any number of things we associate as "natural" interaction mechanisms to trigger something in our interactive experience to happen.
\nI actually didn't know where to start with this. Do people REALLY do brute force tracking of your appendage at every stage and consider every possible outcome of where your appendage could be at any given moment? From what I'd seen, the answer is yes. I posed my question to a private Kinect G+ Community I'm a part of. My question was basically..."How do you even start going about programming a gesture?".
\nBecause this small knit group of people rock - I received C# source code almost immediately for a swipe gesture from a Turkish developer named Furkan Üzümcü (@furkanzmc). I set out to convert it to C++ for my own purposes which I'll share in a bit. It really opened up my eyes to the attention to detail about what you are doing with your body to register a gesture.
\nHere's the anatomy of a swipe right gesture (as performed with your left hand, swiping left to right):
\nSo yeah, I'd call that brute force. Not only are you defining an event flow based on MULTIPLE joints (your hand, elbow, shoulder, head, and torso), but you are approximating allowable "good distance" for things based on the distance between things like your shoulder and torso.
\nThese are important things to realize. This arm motion that sounds so easy, just became a whole body affair. And global/world coordinates really aren't that great to use here since people can be closer to or farther away from the camera and people also come in all shapes and sizes. So, we think of all of our distances as relative to the distance from one body part to the other.
\nWhen I was programming this, I started sitting there, and really watching my body motion. I'd take my arm and start swinging it around...just thinking about how it moved. I felt like Otto the Bus driver from the Simpsons.
\n![\""they](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2013/03/otto.png\")
"they call them fingers but i never see them fing".... "ohh there they go"\nSo, for a swipe left, you can imagine how that works. Just reverse the motion of the left swipe.
Up and down was a little harder. With the horizontal swipes, we can imagine a user as needing to vertically center the gesture just above their torso. But what of vertical swipes? I could easily swipe up at the left, right, or dead center of my body. Any of these are valid in my opinion, and no starting horizontal position of the swipe invalidates the gesture.
\nSo, I took a page out of the relative distance between joints game. I said: "OK - the distance between your left and right hip is the maximum amount of horizontal variance you're allowed to have in your vertical swipe". So if your hand starts below your hip, and your hand is below your elbow, and continues up to where your hand is above your elbow in the right amount of time and with less than the maximum horizontal variance, then you have a swipe up!
\nSame with swipe down, but reverse of course.
\nInteresting thing when you listen for both events, though - if you want to do either a swipe up or swipe down, you make a conscious effort to put your hand into position. To get it into this position, it seems like a lot of the time, you are causing an accidental swipe up! I haven't resolved this issue yet, but it's an interesting one.
\nLots of things to consider here. I also did one on my own - a wave gesture. As in a greeting - "hello", "goodbye" - you know, a hand wave. Here, I simply detected if the hand was above the hip, below the head, and above the elbow. That's my starting position. Then if the hand's horizontal position goes left then right in a cycle 6 times where each motion takes less than .2 seconds, we have a wave.
\nOverall, you're considering a lot of different things when designing a gesture. How "natural" it is for your users becomes how intuitive you make it. Have you considered how different people might interpret a swipe? If a user thinks perhaps a swipe takes place above their head, will they become frustrated if you don't consider this fact? If a user waves super slow, and each side to side motion takes .3 seconds instead of my .2 seconds as designed, is THAT OK? Visual feedback can help greatly, but I think that considering all of the edge cases can greatly increase the natural feel and intuitiveness of any gestures you design even before thinking about visual feedback.
\nAs I said, I'm funneling all this effort into my Node.js plugin. It's probably in the alpha stage right now, so I'll just link you to my gesture scripts over on Github:
\nhttps://github.com/bengfarrell/nuimotion/blob/master/src/gestures/Swipe.cpp
\nhttps://github.com/bengfarrell/nuimotion/blob/master/src/gestures/Wave.cpp
\nI think as my add-on gets more solid, I'll definitely be cleaning these up as well...organizing code better, removing redundancy, etc. This should just give you a good idea of the logic behind the gesture. Thanks again to Furkan Üzümcü for the C# Swipe code that my swipe is largely based on.
\n", "date_published": "2013-03-20T00:00:00Z" } , { "id": "https://benfarrell.com/blog/2013-03-18-lets-kill-javascript-with-sensational-headlines/", "url": "https://benfarrell.com/blog/2013-03-18-lets-kill-javascript-with-sensational-headlines/", "title": "Let's Kill Javascript with Sensational Headlines!", "content_html": "The following article came across my G+ feed today:
\nhttp://techcrunch.com/2013/03/15/the-future-of-javascript/
\nBasically, the article's headline indicates a major throwdown is ahead of us. People will reject Javascript outright... killing it, or we all open our hearts to JS making it the ultimate one for all language. Thankfully, the content of the article is a little better, describing the growing pains as people have adopted the language more and more and speculating how it (and we) will adapt.
\nTo me, though, this and other sensational headlines like it is the same old history repeating itself. Did we really think that the big buzzword that is HTML5 would last forever? Sorry to tell you guys this, but people are getting bored. And when people get bored, the other camp that doesn't like technology X comes out with pitchforks with the great big "I told you so's".
\nIf the pitchfork dudes can sway opinion, then the snowball rolls and gets bigger....and bigger...and bigger.
\nBut whatever shall replace this hole in our hearts, now that technology X has died? There's always something, and when technology Y is announced, it's imagined as a heroic knight riding in on a white horse to save the day. Once that happens, technology X has "died".
\nDig the grave, say your last piece and pull up the hearse.
\nIt will happen to Javascript and HTML5, too. Does it matter, though? What happens to a technology when it "dies"? Lets ask Adobe!
\nFlash is dead, right? Nah - just fallen out of favor in the larger context. Adobe still invests resources into making Flash better and better. They've remarketed it towards games now, but it still does everything it did before. Yah, so it doesn't do mobile without the help of it's little brother AIR.
\nIs it dead to you because it doesn't do mobile? Guess what genius, there are still plenty of laptops and desktops. There are still people consuming videos and games and rich media. Yes, some of this is being eaten by HTML5 adoption. What are we seeing after Flash has died? Well, it simply declined. Some folks moved on. Others didn't. I still hear of plenty of Flash and Flex projects going on, it's just not a compelling story anymore because there's been a decline and headline writers are bored with it.
\nOK....maybe it's too soon. Maybe death doesn't happen that quickly and my example is bunk. How about another Adobe product? Way back in my career's beginning, I was in love with a multimedia authoring tool called Director. That was actually proclaimed dead when Flash started gaining traction. So.....2002 or 2003, maybe?
\nKnow what Adobe just released? Yep, Director 12. That's like 20 years worth of Director by Macromedia then Adobe. Do people still use it much? Nope. Is it still alive? Yep.
\nThere will always be old projects to update, workflows to continue, but if the new hotness comes along - yes, your favorite technology will decline in popular favor.
\nHow bout another example? There was a time when people said Java will take over the world. I wasn't a Java dev until recently, but if you got caught up in the moment, you might seriously believe it would. I was kinda guilty of thinking the same thing of Flash back in it's prime. Tons of Java jobs sprung up, entire enterprises were taken over, it was injected into college curriculum as the real world thing you'll need.
\nAnddddddddd....it declined. But shit, Java in it's decline was probably better than most techs at their peaks. As a result, Java is still there. We still need developers for it, and Google bumped it up a little with Android development. Some people might say that Google brought it back from the dead. It was never truly dead. It just lost several percentage points in market favor.
\nSee a pattern? I do. I've lived it. And I've freaked out when my favorite technology was being threatened thinking I'd die in a gutter doing Flash and Director work for half eaten sandwiches out of a dumpster.
\nSo, Javascript developers, HTML5 developers - listen up. Your favorite things will all DIE! But that doesn't mean that they won't lead perfectly normal lives after death. Javascript might do as well as Java so long as browsers exist as they do today. But just like Java, Javascript will die when marketing people get bored and try to sell you the new hotness.
\nHaving said that, don't use my advice for complacency. Typically when things die, you'll still find work, but you'll get paid less and gigs might be a little harder to find. So don't get tied in to one tech. Explore and branch out. The biggest mistake I ever made, not ONCE, but TWICE was to think that my favorite tech was the bees knees and it would take over the world. I'm seeing it happen with front-enders today. In fact, I can recall one brain melting moment at a meeting when someone suggested that native mobile app development in Objective C and Android was the "Flash" of the mobile generation. It's a detriment, when soon folks will see the light and we'll all be doing HTML5.
\nNo. Cross-platform never wins. It always takes the dive. Ride it high while it's here but don't expect it to last. Just simply get curious about different tools and languages, try them out - and pay attention to the market. Nothing you do will be a waste of time. Anything that dies on you will still be useful, if not in it's current incarnation, then through the concepts and ideas it carries over to all other things in our space.
\nGuess what's living now? All of it. I can be a Flash developer, a Java developer, a COBOL developer, and a Javascript developer. I can make awesome HTML5 experience and make some cool Hypercard experiences (I think). The more technologies that "die" the more opportunities we have to do new and cool things, not the same amount of opportunities or less.
\n", "date_published": "2013-03-18T00:00:00Z" } , { "id": "https://benfarrell.com/blog/2013-03-05-book-review-of-instant-android-fragmentation-how-to/", "url": "https://benfarrell.com/blog/2013-03-05-book-review-of-instant-android-fragmentation-how-to/", "title": "Book Review of Instant Android Fragmentation How-To", "content_html": "Yep, another book review! This one was a shorter one, and sounded interesting, so it was easy to dive right in. It didn't cover what I thought it would, though if you read the table of contents, it's pretty clear what you're getting. The book is Instant Android Fragmentation How-To by Gianluca Pacchiella.
\nI was expecting more on UI concerns from the title, but this book covers four very specific topics: the compatibility package, fragments, the loader (for HTTP operations), and the ActionBar.
\nThe most interesting to me was the good explanation of the loader and the use of threads. I had to deal with this before for loading JSON data, and kinda dove in. It would have been nice to have this book months ago! Anyway - the author Gianluca Pacchiella does a great job explaining the UI thread and what not to do to lock up your app.
\nI also appreciated the explanation and dive into ActionBarSherlock. I had only heard about this before as it's mentioned in various tutorials, and didn't realize it was a nice open-source Android 4 component that allows us to reach back to Gingerbread. "Instant Android Fragmentation How-To" gets very code and project heavy with this and the compatibility package. So - when sitting down with these sections, make sure to pick up your laptop, or at least avoid the e-book and skip right to the website of PDF where the code is legible.
\nI was a little confused with the end of the book - Gianluca just drops you off at the ActionBar and doesn't summarize or wrap up to finish off the book. I was convinced I had a corrupted e-book, but nope...it just ends.
\nAnyway this 66 page book is a good quick read, with some good code examples to grab - just make absolutely sure that you're interested in those 4 topics listed in the table of contents, or you'll be disappointed!
\n", "date_published": "2013-03-05T00:00:00Z" } , { "id": "https://benfarrell.com/blog/2013-02-25-thoughts-on-the-myo/", "url": "https://benfarrell.com/blog/2013-02-25-thoughts-on-the-myo/", "title": "Thoughts on the MYO", "content_html": "Holy jumpin' gestures, Batman!
\nThe MYO just announced themselves on the scene today. It looks crazy good. The website claims that it intercepts the electrical signals coursing through your forearm to pick up on what you will do.
\n![\"two_rings\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2013/02/two_rings.png\")
Read that again....WILL do. They claim it might actually be a little faster than what you actually do in real life with your actual appendage.
\nI did have one thing on their website that sat with me the wrong way, though.
\nThey say "Wave goodbye to camera based gesture control".
\nNow, as amazing as this thing looks to be, there's a storm a brewin'. Gather round and I'll tell you all about it.
\nThe first gesture/motion consumer device that I can think of is the Sony Playstation Eye Toy. It's camera based, and guess what, it's still used. Sony added some glowy balls to it so the camera could see better and called it the "Move". With the PS4, Sony is getting rid of the glowy balls and making the controller all glowy.
\nOf course, the thing that the MYO is really taking a shot at bringing down is the Kinect. This is a camera based system, but adds depth with some infrared rays that bounce all over your room, like sonar. It overcomes the Kinect by being able to operate in total darkness - but the sun (with it's flood of infrared radiation) is total kryptonite for it.
\nBut the big thing about the Kinect is that it's spawning innovation. The OpenNI foundation sprung up and not only provides device drivers for other Kinect like devices, but provides middleware for all. So whether you use a Kinect, a Carmine, or an Asus Xtion, you can work with gestures and skeletal data. In fact, Primesense, the company behind the original MS Kinect tech is supposedly shipping bulk orders of it's teeny tiny sensor to mobile device manufacturers.
\nCamera's ain't dead - they're just starting. Couple this with the Leap that will be coming out shortly, and most likely a metric crap ton of sensors that will make their way to the market in the next few years.
\nWe're venturing into "using the right tool for the right job territory". RGBD cameras (that's red, green, blue, depth) are great for tracking your whole skeleton. They are great for tracking multiple users at once. They are better and worse then RGB cameras for being able to operate in the dark, and NOT operate in the sun respectively. The Leap will be better than all-purpose RGBD cameras for their ability to focus in and capture the complexities of your fingers, though there is OpenNI middleware that does just that.
\nThe MYO sounds amazing and it will be nice having a good one on one connection for your gestures - so long as you limit what you want to capture to your arms and hands and limit the number of users to the number of devices you want to purchase.
\nLike I said, there's a storm a brewin', and we're gonna see a huge gesture and motion tracking market with some overlap, sure - but I think the MYO and RGBD cameras will definitely be able to stand on their own just fine. I have a Kinect, an Asus Xtion, a Leap on pre-order, and plan to get a MYO when they get released. There will be correct situations for all, and all will be a blast to play with.
\n", "date_published": "2013-02-25T00:00:00Z" } , { "id": "https://benfarrell.com/blog/2013-02-24-book-review-of-android-4-new-features-for-application-development/", "url": "https://benfarrell.com/blog/2013-02-24-book-review-of-android-4-new-features-for-application-development/", "title": "Book Review of Android 4: New Features for Application Development", "content_html": "I don't normally do book reviews, but I just happened to be working on an Android app and trying to build up my UI skills on the platform, when a Packt Publishing representative reached out to me to offer a free book in exchange for an honest review.
\nThe book was "Android 4: New Features for Application Development" by Murat Aydin. Before I talk specifically about THIS book, I should say that this is my third or so exposure to reading Packt published books. I've leafed through a couple of Joseph Labreque's Flash and AIR "New Feature" books.
\nIt seems Packt style books follow a specific style. They're on the shorter side, Murat's was 166 pages, crammed full of code examples. They also seem a little dry. When I say dry, I mean I don't really get a sense of the author's opinion's and feelings on a subject - just a straightforward laundry list of features briefly explained followed by an extremely detailed code example.
\nIs this good or bad? Well, I think it depends on how you read it. When I read in bed on my Kindle, the book feels pretty light and flimsy. When I read a tech book, personally, I like to put all the code examples aside and get a good mental picture of what the author is trying to express. I end up glossing over all the code examples, and just trying to break out what the author is trying to convey in my head and sleep on it.
\nWith "Android 4: New Features for Application Development", like I said - I just happened to be making an app at the same time. So when I wasn't on my Kindle, I was using the Packt book reader (packtlib) available on their website. I was pleasantly surprised to see that reading it this way was actually really nice.
\nMurat covered a couple new features that were immediately useful to my project: the ActionBar and Fragments. When implementing an ActionBar, it was nice to copy and paste code from his project to mine and get quick results. I would have liked to have seen more of a diversity of situations - in my application, there were several situations (like when I theme my app) where the ActionBar doesn't work correctly. Murat's example app here was good, just a simple one. All in all, though it was a good introduction to the ActionBar so long as you used the code examples yourself to explore.
\nSame with the Fragments. My curiosity was piqued in terms of what Fragments were because I was already using the Google Maps fragment in another project. I didn't quite understand what a Fragment was....I just new I was using one. "Android 4: New Features" gives some good working examples of how to create and use Fragments through code - but I just didn't quite get a clear picture that they were just modular components with their own lifecycle until I looked at the Google Android Developer's documentation.
\nSo I think that's the theme here with how I feel about this book. I want to hear Murat's voice more. I'm betting he has some great thoughts and opinions and I want to hear them. I want to hear, in his own words, what these features are and what they mean to him. I want something to read in bed without my laptop that I can get a good mental model of what they author is trying to convey.
\nWhat I got from this book was something I could sit down with at my laptop and copy very thorough and solid code examples with a little bit of guidance from Murat. I think that's probably great for some people, just not my cup of tea with my "dive in headfirst attitude" where I skip the simple stuff and jump into making my own app.
\nFinal word:
\nIf you want to read in bed and simmer on the concepts - then I can't recommend it.
\nHowever, if you're familiar with the Packt publishing type of book and you like it, "Android 4: New features for Application Development" is a good one. If you're not familiar with Packt - ask yourself if you want constant handholding through some basic, thoughtful code examples. If yes and you want to familiarize yourself with Android 4 - again, a pretty good book.
\n", "date_published": "2013-02-24T00:00:00Z" } , { "id": "https://benfarrell.com/blog/2013-02-20-the-upright-spass-a-javascript-instrument-in-thin-air/", "url": "https://benfarrell.com/blog/2013-02-20-the-upright-spass-a-javascript-instrument-in-thin-air/", "title": "The Upright Spass - A Javascript Instrument in Thin Air", "content_html": "Well, well, well....
\nAll my talk of OpenNI, C++, NodeJS, etc in recent months was pretty much all boring until you put it into practice and make something cool.
\nI did just that....well, I think it's cool. And just plain weird, really. Here's a motion controlled instrument I made that's Javascript through and through. It's Node.js at the heart, with a HTML/Javascript display. And yah - I snuck in some C++ to wrap the ever awesome OpenNI SDK.
\nI present to you....the "Upright Spass":
\nI've played around several months ago with the Kinect SDK playing a keyboard in thin air. What I was playing with then was Windows only, Kinect only, and need Adobe AIR to route things to websockets for the browser.
\nSo using my new found powers over the past few months with:
\n....I now have a nice little handtracking utility that runs in Node.js using OpenNI and NiTE to power my skeleton tracking.
\nI didn't care for the horizontal layout of my old virtual piano - so I inverted the axis, and made the instrument control upright. Hence - "Upright Spass"....the anti-bass, the bass that is not a bass, just empty space.
\nWhat was also crazy hard was producing decent sound with Javascript. I don't care what language you do this in, creating sounds from scratch is hard. You could go years studying and tweaking new sounds to match what already exists in the world.
\nSo to solve this? MIDI. Hell yes, MIDI! I found a nice robust Node.js MIDI addon. So instead of making my own sounds banks, I send it out over my E-MU MIDI USB controller to my Korg X3 keyboard
\n![\"x3\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2013/02/x3.jpg\")
And wow....the site I grabbed this image from is calling this keyboard (made in 1993) "vintage". I feel old, damn.
\nAnyway - I'm running Ubuntu for this whole operation, so to route the MIDI from Node.js to my keyboard, I used Jack. Jack offers you a nice little audio server. You can patch in your MIDI through out to the E-MU MIDI USB device in. Voila, start make the link and start the Jack server.
\nSo, I got this motion controlled midi thing all rigged up, and it's REALLY hard to play. There were a few problems:
\nEven after solving a few of these problems, the Upright Spass is really hard to play. My performance was pretty much a disaster - but maybe I can tweak and practice and get passable at it.
\nMy code for this is up on github. I mentioned the link for my Node.js AddOn previously - that's here:
\nhttps://github.com/bengfarrell/node-sweatintotheweb
\nand this particular project, the Upright Spass, is here:
\nhttps://github.com/bengfarrell/uprightSpass
\n", "date_published": "2013-02-20T00:00:00Z" } , { "id": "https://benfarrell.com/blog/2013-02-01-i-wrote-my-dad-a-letter-with-the-webspeech-api/", "url": "https://benfarrell.com/blog/2013-02-01-i-wrote-my-dad-a-letter-with-the-webspeech-api/", "title": "I wrote MY dad a letter with the WebSpeech API", "content_html": "Sometimes technology can be so exciting, but then turn out so....meh.
\nI was extremely excited to checkout the brand new WebSpeech API in Chrome 25 Beta. The promise of talking to your computer on a webpage sounds freaking awesome - especially after I saw this demo:
\n\n\nPretty cool, right? So I decided to send my own dad a letter.
\nThe demo didn't turn out...the best. I've actually never downloaded a Chrome beta before, and thought I needed to download Chrome Canary (v26) to try out this upcoming feature. That ALSO didn't work the best - plus the browser crashed after I finished most recordings.
\nBefore recording the letter to my dad though, I spent some time actually working with the speech API, and despite how well it DIDN'T work, it was quite easy to use.
\nFirst you initialize and set a couple of parameters:
\nvar spx = new webkitSpeechRecognition();\nspx.continuous = true;\nspx.interimResults = true;\n\nThe continuous value basically seems to mean that as you speak more, it continually corrects the output based on context. And interim results mean that it will give you spur of the moment "What we think you said" even though you haven't finished speaking yet.
\nI then set up some event handlers, and start...
\n\nspx.onstart = function() {\nconsole.log("Start");\nfinal_transcript = "";\ninterim_transcript = "";\n}\n\nspx.onresult = function(event) {\nfor (var i = event.resultIndex; i < event.results.length; ++i) {\nif (event.results[i].isFinal) {\nfinal_transcript += event.results[i][0].transcript;\n} else {\ninterim_transcript += event.results[i][0].transcript;\n}\n}\n\n}\n\nspx.onerror = function(event) {\nconsole.log(event)\n}\n\nspx.onend = function() {\nconsole.log("End");\nconsole.log(final_transcript);\nconsole.log(interim_transcript);\n}\n\nspx.start();\n\n\nIt functioned quite well in theory. I'd start it up, as I was speaking, it would send out my interim result and final results via a JSON object (which was kinda complex to look at, but I can understand the complexity if they are giving all of this info).
\nBut the results were...again....meh. It only understood me 10% of the time, and of those times I had to speak verrrrry slowly. The other times it seemed to have results that rhymed or skipped words or just plain didn't even come close.
\nGoogle Chrome's implementation doesn't even do anything locally - instead it sends out info about your speech over the web, processes it on their servers, and sends it right back to you. One would assume that it works the same way as the AWESOME MAGIC that is Google Now on Android phones/tablets. Seriously, THAT speech input rocks my world. I wouldn't have had such high hopes for Chrome 25 if Android didn't work so crazy well.
\nPerhaps it's a microphone difference? I had mine cranked and test levels prior to calling it a day. Maybe it's noise/echo cancellation on the phone? I don't know - but on my Windows 8 box with the built in Lenova Ideapad mic (which sounds fine to me), it just didn't work out.
\nMy dad's response, BTW, was only one word:
\n_Huh???
\n-----Original Message----- From: Benjamin Farrell To: My dad Sent: Thu, Jan 31, 2013 8:40 pm Subject: hi dad
\nKeeps getting better hi dad email area now I have no excuse to not raining Solis_
\nI was hoping for a stronger close to this little gag, but I guess that will suffice. Another "meh", just like the WebSpeech API.
\n", "date_published": "2013-02-01T00:00:00Z" } , { "id": "https://benfarrell.com/blog/2013-01-31-android-mediaplayer-you-are-too-co-dependent/", "url": "https://benfarrell.com/blog/2013-01-31-android-mediaplayer-you-are-too-co-dependent/", "title": "Android MediaPlayer, You are too Co-Dependent!", "content_html": "I actually really do enjoy Android development - it's pretty smooth. Got me some Intellij IDEA as my work environment and I feel like a damn Viking Captain....or an astronaut....or something else really cool. I like Java as well. It's very reminiscent of my old Flash career, and the XML based layout markup is ever so much like Flex. And HOLY HELL, the design mode of these layout editors actually work great in Intellij IDEA. I could never say the same for Flex!
\nContrast that to iOS, which is still nice, but you're using visual editors, can't edit the underlying stuff, and Objective C where I don't have garbage collection....
\nAnyway, I digress - I came here to complain.
\nAnd what I complain about doesn't seem to be any one platform or technology or language, it's about how messed up video playback is in that one platform, technology, or language. I've been through it with Flash (which is pretty nice actually), iOS, HTML5, and lately: Android.
\nWhat's messed up in Android isn't the missing documentation or needed API's to get things working - it's that they throw everything at you and it only works sometime on various devices! Some would say the F-word (f-r-a-g-m-e-n-t-a-t-i-on). I dare not speak it here because I don't buy into the end of the world nay-sayers that say "Android is sooooooo fragmented, use a different platform". Unfortunately though, in regards to video, it seems to be that way.
\nIt's fragmented in a big way from Gingerbread to Honeycomb to Ice Cream Sandwich in regards to native streaming video via HLS. Gingerbread doesn't support it at all - Honeycomb, it probably will, and yes it's there in ICS. So Apple's streaming video tech that it has successfully pushed across our entire industry may or may not be available on your device...natively that is. Some smart folks though, have made C++ libraries to handle it across all versions, and I'm sure they get paid handsomely for it (by you, the developer).
\nThat's cool, though, maybe you don't need to stream - maybe you just want to play a short 5 minute or so piece of progressive content. That's actually FINE, but be careful how you do it.
\n\nAt the very base level of media playback in Android, we have the MediaPlayer. This little guy plays video, audio, whatever. Interesting thing, though, is that it doesn't SHOW you the video. To actually see what's playing you need to call mymediaplayer.setDisplay(mysurface). So I've been using a "SurfaceView". You can probably imagine what normal views are - they hold graphics and stuff for display. You could use a Layout (like a FrameLayout or a LinearLayout) to bring order to your chaotic mess of graphics on your view - but that's really what a view does, it holds graphics.
\nA SurfaceView is weird, though. Sometimes you need hardware accelerated graphics power at your disposal. So what SurfaceView does, is it punches a hole in your device right down to the GPU and uses that area to accelerate your graphics - instead of the normal layering that usually happens. It looks like graphics that are layered with it are handled pretty nicely - I'd chalk that up to Android/Google engineering attention to detail (or something).
\nAnyway, that's what the SurfaceView is - its a hole punched through your graphic layers right down to the GPU. Its commonly used for 3D and video. Since we want our videos running smooth as butter, we are strongarmed into using the SurfaceView.
\nSo, there it is - you need to set your video path or URL, attach it to the SurfaceView with setDisplay, set the audio stream with mymediaplayer.setAudioStreamType(AudioManager.STREAM_MUSIC), then prepare.
\nActually, don't do prepare - do mymediaplayer.prepareASync(). Prepare locks up the graphics thread until your media starts playing. You can then listen for the MediaPlayer onPrepare event and run mymediaplayer.start() to kick off playback.
\nMeanwhile, there's a WHOLE other situation going on with being EXTREMELY CAREFUL what you do with the MediaPlayer. So complicated, Google has apparently given us a state diagram....
\nIt's not that I think diagrams are too complicated - but this one is just really important. If you call getCurrentPosition, or pause, or resume on the player and it's in the wrong state......wellllllll your app crashes. Even worse, the media player won't actually tell you what state its in. There's no "getState" method or property. Nope...how do you know? Well, Google suggests you keep track of it your DAMN self with the appropriate listeners.
\n\nThink the above is complicated? Well, good thing there's VideoView. This View is a MediaPlayer wrapped in a SurfaceView for the ultimate convenience. It passes on some of the more important MediaPlayer events like onPrepared, onComplete, and more. And then if you want more - well, grab a reference to the MediaPlayer from the onPrepared callback. It's the first parameter. But again, be very careful to not call something on your player when it's not in the correct state!
\nAnyway, it's pretty simple to use - use the setVideoURI() and start() methods to get things going. Just like any other view, it'll size nice and dynamically along with your layout. You can even setMediaController() on it to attach a native Android UI media controller.
\nSo, yah - MediaPlayer is pretty co-dependent. But that's in a good way, right? Why bother with MediaPlayer when we can just use VideoView?
\n\nSo, we actually had a use case in our app, where we need to hide the video momentarily before bringing it up again. You can do some pretty sweet things with the SurfaceHolder callbacks - and listen for when the SurfaceView is created, destroyed or changed. Unfortunately, those seem to do NOTHING in the VideoView. Yah, you still get the events, and you can listen for whatever you want - but your video always seems to shut down when the VideoView is made to go away.
\nAlso unfortunately, is when looking at the source code, methods are mostly private. So there's not even any hope of overriding something and doing your own thing.
\nOK - well, I need to hide and bring the video back. VideoView can't do that. Well, let's rip that co-dependent bastard called MediaPlayer right out of there and implement it ourselves.
\nYah - I had problems with that too. To play multiple videos sequentially with my custom View, I had to set the view to gone before loading the video, and then set it to visible on prepare. Without this, I'd get some pretty crazy errors most of the time.
\nI'd get the same crazy errors rarely when running my video, removing its view, then reattaching again. Most of the time it worked perfectly - I just listened for surfaceCreated, then attach the running media player again with setDisplay. Other times, it would error out when I did that and I'd get that crazy error.
\nWhat's the crazy error? The error is 1, which means it's unknown, and the extra is - 2147483648. I can't find any information whatsoever on that error online that makes sense. Some folks on stack overflow say the file has a corrupted header size on the server, others claim its an invalid format, and more. None of them complain about this happening in the freaking middle of playback.
\nOk, well, whatever, it happens rarely, and we can catch it and move on. Fun fact: if you catch the error, you can return false from your custom error handler to have the MediaPlayer mark itself complete and have the onComplete handler called.
\nSo I have a custom media player that seems to work well MOST of the time on my two Honeycomb tablets. When it fails, it's rare, and we can move on....PROBLEM SOLVED!
\n....that is until I try it on my Kindle Fire. I can hear the audio but not see it, while on the other devices it works perfectly. Seriously, what the hell? I spend mucho amounts of time comparing my code to the Android VideoView source and don't see anything different - yet still, here we are. And now, I'm willing to bet why VideoView doesn't give you much option to keep this alive in the background - because it's just a quirky mess on some devices.
\n\nI never did figure out my problem. In the end, I figured we'd better just use the VideoView and not have it removed from the screen in mid-playback. Don't do anything fancy with your surfaces because it'll probably fail on some device somewhere. Perhaps someone who is smarter than I knows where I fell short. Or perhaps, this is an insurmountable problem. It sure seems quirky as hell to me, though!
\nOther issues included playback of some mp4's at all on some devices. Read up on the supported media formats. Most devices seem to be very forgiving allowing you to encode a bit differently as long as it's in wide practice, but some older Android models don't like some of these encodings - so it's safer to stick to the speck.
\nAnd so, that is why the MediaPlayer is way too co-dependent on the VideoView. Cause you think MediaPlayer can operate without its buddy like that. It tricks you....lulls you into a false security, but always bites you in the end!
\n", "date_published": "2013-01-31T00:00:00Z" } , { "id": "https://benfarrell.com/blog/2013-01-30-further-adventures-in-openni-and-node-js-making-it-crossplatform/", "url": "https://benfarrell.com/blog/2013-01-30-further-adventures-in-openni-and-node-js-making-it-crossplatform/", "title": "Further Adventures in OpenNI and Node.js - Making it Crossplatform", "content_html": "I hate to do things twice, but sometimes it just needs to be done twice, three times, or more. Luckily after the first time of faking things through, you become a bit of an expert at the many things that can go wrong!
\nAnd so it goes with building out our Node.js OpenNI plugin.
\nJust a couple weeks back, I posted on building a Node.js plugin to use gesture and joint data provided by the ever fantastic OpenNI project. For those that aren't familiar, well, it's LIKE the Kinect, but crossplatform and open. I posted my experiences building a couple simple gesture/motion controlled plugins as an Ubuntu user.
\nThere are a couple experiments I'd love to do, but I can't QUITE do them on Ubuntu yet. The first is to check out Viim, a more robust middleware than NiTE. The middleware, in OpenNI land, bridges the gap between the depth and RGB data and the actual gestures and skeletal data.
\nLike I said, Viim seems MUCH more robust than NiTE offering a full suite of gestures and other goodies compared to NiTE's lowly three gestures - though NiTE DOES offer skeletal data. It seems that Viim is on the cusp of being released for Ubuntu, but for now, we must make do with Windows and OSX.
\nAnother little thing I wanted available in my experiments is speech interaction. The OpenNI project doesn't seem to offer this like Microsoft's Kinect SDK. Nevertheless speech interaction is important to anyone studying in the Natural User Interface dojo. Luckily, new to Chrome 25 is the Speech API! v25 isn't quite out yet, but we can grab the Chrome Canary build - which DAMMIT, isn't available on Ubuntu side by side with the production version of Chrome.
\nOh well - it's probably time to try things out on Windows. Even if these things are released tomorrow, I'm not wasting my time. It's good to make sure all my experiments work cross-platform. I want YOU to try this stuff out, whether it be on Windows, Linux, or whatever!
\nBeing the noob that I am - C++ compilation on Linux was brand new to me. So GCC and G++ were new and scary. But using Make wrapped things up into a nice little command line package. It was easy to just type "make" on the command line and have everything just......go. Likewise, with Node.js' build tool: node-gyp. Once I had my build.gyp file setup correctly, it was easy to just run "node-gyp configure build".
\nGyp would create the appropriate Make files with the configure command - and then use G++ to build the stuff that the "configure" command spits out.
\nTurns out that Windows was surprisingly similar, with one curveball! Node-gyp on Windows spits out "vcxproj" files (and friends). These files are actually Microsoft Visual Studio project files. So, you COULD open these right up in Visual Studio if you wanted to. I wanted to see if we could still run these on the command prompt - the same "node-gyp configure build" routine, you know?
\nWell, aside from making sure we have Node-gyp installed from the Node Package Manager (npm) and Python installed to complement Node-gyp, we'll need some Windows tools:
\n\nPlease note that I'm using Windows 8, so 2012 works for me! Your mileage may vary. And because I'm using Windows 8, I had trouble with my next dependency: OpenNI and NiTE!
\nWith my old copy of my OpenNI 2.0, I actually couldn't get things compiling on Windows 8. One of the header files complained that my C++ compiler was too new. Luckily I didn't have to put too much brainpower in here, because OpenNI 2.1 was just released, and that solves the problem. Visual Studio 2012 happily updated the project files provided by the samples, and I could create executables, so all was quite well there. Downloading and installing NiTE appeared not to have similar problems.
\nAfter some trial and error, I was able to figure out the secret sauce. I've included full instructions in my Readme file on Github. But, what I ended up doing, was taking the files from "C:/Program Files/OpenNI2/Redist and dropping them at the root of my module. This included some DLL's, lib files, and more. Basically you just need the libs and DLLs, though. I also copied NiTE2.dll from my "C:/Program Files/Primesense/NiTE2/Redist/" folder to get the NiTE middleware working.
\nI also ended up changing the link paths and ditching one of the compile options on Windows. While the "-Wl,-rpath ./" was the secret code to add to our Gyp file to make it build on Linux, this flag doesn't work at all on Windows - we'll just leave it in and Windows warns us and moves on. It seems that all Windows needs is the correct path in the "-l" flag. Linux needed a little love with those other extra options, but Windows performs like a champ with just -l./pathto/OpenNI. So, in my Gyp file, I just created some variables that are set depending on which OS you have, to point to the correct path.
\nLast step was compiling! Don't use the normal DOS command prompt though - load up the Visual Studio SDK command prompt. Navigate to the source of the project and do "node-gyp configure build".
\nVoila! You've built an OpenNI/Node.js plugin - on Windows this time!
\nMy source for this module is here, and a simple usage example (which I describe in depth on my first post) is here.
\nGood luck and happy gesturing!
\n", "date_published": "2013-01-30T00:00:00Z" } , { "id": "https://benfarrell.com/blog/2013-01-16-jsconf-and-cfobjective-2013/", "url": "https://benfarrell.com/blog/2013-01-16-jsconf-and-cfobjective-2013/", "title": "JSConf and cfObjective 2013", "content_html": "Guys and gals, I'm incredibly honored to be among the select group of folks speaking at not one, but TWO conferences in May. I think JSConf is pretty self explanatory - it's a Javascript conference. I shouldn't say this, as I really haven't been to many Javascript conferences, but it's probably the best damn JS conference in the country. That's what I hear anyway!
\ncfObjective is a little more of a head scratcher - you might have never heard of it if you're not using CF! I'm privileged to know some awesome and smart folks who love working with Adobe Coldfusion. To be honest though, CF was never my cup of tea - nor was the server side for that matter. However, I was fortunate enough to speak at cfObjective last year on their brand new "jsObjective" track. The track is just peppered silly with some great minds talking about HTML/CSS/and Javascript - and I'm very lucky to be one of them again this year. If CF IS your thing, well, there is no better Coldfusion conference, is there?!!
\nIt's just plain awesome to be speaking in two conferences such as these - let alone 2 weeks apart. I'd say I feel like a rockstar....but that notion peaked in our industry, didn't it? I feel like the damn belle of the ball.
\nAnyway, I was looking over my topics on both sites beaming and throwing dollar bills in the air - and I thought to myself "These topic descriptions are technically true, and the most succinct way to write them up, but I just don't know if I'm conveying what the audience will be getting into".
\nSo, let me clear the air! I've been obsessed with two things since last fall - Google Ingress and Natural User Interfaces. I'm obsessed with Natural User Interfaces because they combine user interfaces, graphics, rocket science programming, and cool gadgets. Now my topic description says you'll learn about the Kinect and Javascript. The Kinect part is as true as a Kleenex is a tissue.
\nPeople are excited about the Kinect and know what the Kinect does. When you say Kinect, many will have an understanding of what that is, and BASICALLY what it can do. What if I said I was going to talk about an Xtion? Or a Carmine? Both are pieces of hardware just like the Kinect, but you've never heard of them. And guess what? They work on Windows, OSX, or Linux.
\nWhen I say we're going to learn about the Kinect and Javascript - I don't mean JUST the Kinect, I mean the whole motion sensor and natural interaction ecosystem. I want to bring that ecosystem to my other obsession - web development. And you know, when we talk about web development - we're talking cross-platform. So my presentation isn't worth a dime, UNLESS we're talking cross-platform.
\nSo, yes - we'll get gestures and hands and your entire skeleton tracking in the browser. I'll tell you briefly how you can get it there. I fancy Node.js at the moment to do my bidding. But really, I'll give you the big picture on what you can do, how it all works, and fill you in on some nifty browser experiments!
\n", "date_published": "2013-01-16T00:00:00Z" } , { "id": "https://benfarrell.com/blog/2013-01-15-using-a-kinect-like-device-in-node-js-with-openni/", "url": "https://benfarrell.com/blog/2013-01-15-using-a-kinect-like-device-in-node-js-with-openni/", "title": "Using a Kinect-like Device in Node.js with OpenNI", "content_html": "Back in December, I started playing with OpenNI. Later in the month, after I posted that first blog post about getting my feet wet, OpenNI launched version 2.0 of their software. This meant a few huge things, but first I'll recap what I'm trying to accomplish!
\nI've been OBSESSED with bringing motion detection to the web. Websockets are probably the best way to bring it to the browser right now - with some kind of backend that deals with the depth camera. I've explored projects like KinectJS which do it this way and do it well. But, the problem is that despite KinectJS being the best tool for the job so far, it relies on both Adobe AIR, Windows, and a Microsoft Kinect.
\nWhile none of those are BAD things - I'd prefer to open it up a bit, especially since I'm running Ubuntu.
\nAll this is old news though - and I wrote about it already. I was already using the OpenNI Java SDK to ooooh and aaaah over my live depth camera. When OpenNI 2.0 dropped in mid-december I had my Kinect clutched in my hot hands and ready to hack!
\nThere were a couple problems with this, though!
\n\nThe first thing is that the Kinect is no longer supported where Microsoft doesn't officially support it. This means no Ubuntu and no OSX. If you're using a Kinect, OpenNI uses the Kinect SDK as the device driver. It makes sense, though - you're actually not LEGALLY allowed to use the Kinect SDK unless you have the official Kinect for Windows device which costs over $200.
\nPreviously, I had been using a cheap XBox Kinect with the Kinect SDK on my non-Windows system through OpenNI. It was great while it lasted, but it really wasn't the right thing for OpenNI to do - they were stepping on MS's toes a little bit allowing this to happen.
\nSo, now that OpenNI is doing the right thing, I had to face facts that I could no longer use my cheapo XBox Kinect. Instead, I hopped on to NewEgg.com and bought an Asus Xtion Pro Live. This thing is pretty cool - it seems equivalent in every way to the Kinect, except it doesn't have a motor so that it's head can move around and track you. It's smaller too!
\n![\"My](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2013/01/IMG_20130104_174634.jpg\")
I should note that I found out later that it may be possible to use the Kinect with OpenNI on OSX and Ubuntu with a separate project called Freenect. But you know what? I'm happy being on the up and up now!
\nOpenNI turned out to be super easy to install. I felt like I had to jump through a few hoops and went through a few frustrations trying to get OpenNI 1.x working. But with 2.0, I had the samples running in a matter of minutes. Woot!
\nThey divided OpenNI up into two pieces with clear definitions. OpenNI is for freely available device drivers and low level depth camera info. So if you load up the OpenNI samples, you can see a depth stream like so:
\n![\"NI](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2013/01/sampleviewer.png\")
NI Viewer Application
So, if OpenNI is the "heart", a project called "NiTE" is the "brains". NiTE is middleware - its a library that takes the depth data from OpenNI and does what you'd normally think a Kinect would do and recognize user gestures, posture, skeletal/motion data, and more. I don't believe the source code is release for NiTE like it is for OpenNI - you can find OpenNI on Github. For those familiar with OpenNI/NiTE 1.x - the separation of tasks that these two provided have definitely been more segmented and defined in 2.0. Now there's really no question what each does, functionality-wise.
\n\nBut, when I was all done playing with the samples, I wanted to hack some code - lo and behold, there were NO JAVA FILES! Only these strange .cpp files. Luckily, I at least knew enough that this was C++. But my heart sank - I could vaguely work my way around Java due to my Android hacking, but C++ is something I never really tried. Luckily, I pulled enough out of my subconscious to recognize that I could just run "make" on my command prompt in the folder to build the project from scratch.
\nThis was OK - but then it hit me....
\nI really wanted to use Node.js in the FIRST place for this and not Java. Socket based communication is quite popular in Node.js. So now my hand is forced - it's time to write a C++ Node.js AddOn that lets you use OpenNI right in Node and broadcast the Node data over websockets to my page. I was quite giddy at the prospect of all of this being in Javascript, so I set to work.
\nFirst I learned how to make Node.js AddOns. Once I learned how to do that, I struggled GREATLY with the Node.js build system (node-gyp) to include the files in OpenNI shared library files the proper way. After stupid mistakes festered in my brain for days - I finally got it building! And now it was a matter of building things the RIGHT way - with OpenNI AND the NiTE middleware so I could recognize a few gestures.
\n\nActually - I don't know what the average Node.js AddOn looks like - but frankly, I think that when you take into account how OpenNI works you need to get a little advanced with your plugin.
\nWhy? Well, the OpenNI/NiTE examples run in a massive while loop that locks up the entire main thread. This is OK in the example, because it's a self contained system. We don't need to event out, accept input, etc. Basically, in the examples, for each iteration of the while loop, a "frame" is read. In OpenNI terms, a "frame" is what you'd think it is - it's a frame of video - only here's it's a frame of DEPTH video! Each pixel is represented as depth data.
\nWith NiTE, though, each "frame" contains a list of body part/joint positions, gesture data, and whatever you listen to.
\nSo - how do we incorporate these while loops if they are going to block the entire Node.js process?! I had actually tried having Node.js Javascript code call in every 50 or so milliseconds to query the frames. This seemed to have the effect of doing everything fast enough that it caught my gestures - but it seemed a bad solution to my problem.
\nInstead, I learned how to create a separate thread in C++, and using the Node.js C++ thread utilities, I spin up the thread which my while loop runs and processes the frames. If a gesture is found, it messages out to the main Node.js thread to report the event - which I then pass on to my Javascript.
\n\nSo, in the end, my first Node.js AddOn to enable OpenNI/NiTE is a simple one. If you start the plugin, you can wave to the camera or make a clicking motion with your hand. Node.js, with the power of OpenNI/NiTE will send you a string that indicates which gesture you just did! It's a small step, I know, but one I'm super excited about.
\nBest of all, you can either use my stuff as I go along, or use it as a template or learning tool so you don't have to hit your head against the wall as hard as I have trying to get this working! I'd like to get more features very soon, and pipe this stuff through websockets to my page.
\nRight now though, I have a super simple, but WORKING Node.js motion capture AddOn. YEAAAAAH!
\n\nNext I DEFINITELY want to add support for more gestures and skeletal data. Unfortunately, I see that NiTE only seems to support 3 gestures. So, I'm not exactly sure how to continue with that - though I do believe the skeletal data will be pretty straightforward.
\nOn the plus side, middleware is interchangable. A company called Covii has another middleware solution called Viim - and from the looks of things, gestures are VERY thoroughly supported. Only problem is that the solution costs a couple hundred dollars. I'm sure it's QUITE worth it - but I really wanted to show you guys something more approachable (technology-wise and wallet-wise). Anyway, I'll figure something out, even if it means having to plunk down a couple hundred for an awesome piece of middleware.
\nAll of this is well and good, but many probably don't care how the AddOn works, how I made it, or anything else. You might be scared of C++ (I know I was!) and just want to try things out in Node.js. In that case, I do have a github repo for you. For now, you'll have to follow the readme, and make the file yourself using the provided instructions. I'm calling this project "Sweatin' to the Web" and you can grab it here.
\nLike I said, it doesn't do much yet! But, I do think that the frustrating part is over, and the fun begins as I can dive deep into what OpenNI/NiTE and more can do!
\n", "date_published": "2013-01-15T00:00:00Z" } , { "id": "https://benfarrell.com/blog/2013-01-03-c-and-node-js-an-unholy-combination-but-oh-so-right/", "url": "https://benfarrell.com/blog/2013-01-03-c-and-node-js-an-unholy-combination-but-oh-so-right/", "title": "C++ and Node.js: An Unholy Combination....But Oh So Right", "content_html": "Update for May 2016: Please note that this article is now 2 1/2 years old and was done with Node v0.8. Unfortunately that means the methods I described our out of date!
\nI have more complicated plans in the works, but for now I'm taking a few giant leaps into Node.js plugin development to see if it's feasible for my master plans - and it not only looks feasible for a newcomer like me, but really easy in an unsettling way.
\nYou make your first Node.js plugin and think to yourself: "Wow, that's all it was?! Why was I scared of C++ for so long?". And then you sort of wondering if you just hit a big iceberg and you're only seeing the tip. Well, what if I am?
\nI present to you guys the tip of the iceberg.
\nFirst, one needs to realize that Node.js is built on Google's V8 engine - the very same Javascript engine that runs in Chrome. C++ (and NOT Javascript) is it's native language. What's even odder to realize is that when you talk about making plugins for Node - I'm starting to think that I'm ALSO making plugins for Chrome or Chromium. It's just crazy to think about all that power you're about to wield.
\nAlso, Node.js is the PERFECT place to start. I'd absolutely love to make awesome desktop applications in Chromium - with Adobe Brackets and Spotify being some pretty prominent examples of Chromium based applications. But I can only imagine how much more complicated having to deal with a C++ application with a user interface is than just a plain old command line program. Soo....I think Node.js will be pretty perfect here.
\nMy reasons for attempting C++ in Node.js are basically the only reasons anyone ever introduces C++ into something else.
\nIt's fast.
\nI have a C++ library that I'd like to use and I'm not smart enough or don't have the time to rewrite it.
\nBoth of these reasons apply to what I'd like to do, but mostly the latter. So given that, I shoved my C++ library aside and tried to figure out how to do the most trivial C++ module for Node.js. It turns out that a lot of what's on the internet is wrong depending on whether you're running the latest version of Node (0.8) like I am. Many articles you'll find on the internet delve deep explaining how to use the WAF tool to compile C++ to Node.js.
\nUnfortunately, for the two hours I wasted trying to get it working, WAF is deprecated in Node! Node-waf is a Node wrapper for the WAF build tool, but going forward we'll be using GYP...or rather node-gyp.
\n\nTo get node-gyp up and running, install from NPM:
\nnpm install -g node-gyp\nSimple right? Just like you're installing any other node package. Gyp reportedly depends on Python, Make, and a C++ compiler like GCC. Doing this on Ubuntu, I don't think I had to worry about these dependencies - as they were already on my system.
\nAssuming everything is set on your system - the best next step is to check out the official Node.js documentation. I was a little surprised by this, but Node.js seems to have excellent documentation. Officially, these modules/plugins are called "addons" and can be seen in the Addons section of Node's documentation.
\n\nHere's the "Hello World" example copied right from the docs:
\n#include <node.h>\n#include <v8.h>\n\nusing namespace v8;\n\nHandle<Value> Method(const Arguments& args) {\n HandleScope scope;\n return scope.Close(String::New("world"));\n}\n\nvoid init(Handle<Object> target) {\n target->Set(String::NewSymbol("hello"),\n FunctionTemplate::New(Method)->GetFunction());\n}\nNODE_MODULE(hello, init)\nBreaking this down looks daunting, but once you figure out the rhyme and reason, it's not too shabby. I think the includes at the top are fairly self explanatory - but lets start at the bottom. Here we declare the NODE_MODULE. It's called "hello", and it's entry point is "init". So when we call it from node, the module is actually being named "hello" here, and it will run the "init" method right when it loads.
\nThis example is a bit confusing, as the NewSymbol is also called "hello". Ignore this for now - first lets take a look at the "binding.gyp" file (the compiler build instructions):
\n{\n "targets": [\n {\n "target_name": "hello",\n "sources": [ "hello.cc" ]\n }\n ]\n}\nThis target name "hello" has to match the NODE_MODULE declaration "hello". Basically, this "target" targets the NODE_MODULE name, and this NODE_MODULE needs to specify the main entry point of our little AddOn, which is the "init" function. And lastly, we have to specify which file the target is in - and here it's "hello.cc". It's actually a matter of preference in C++ land whether you choose to call your file *.cc or *.cpp (I tried it, either works!)
\nSo, fantastic! We've got ourselves a Node.js AddOn! To compile run:
\nnode-gyp configure build\n...from your command line. It's actually two steps - configure prepares the various files that are needed to make the AddOn, while build actually does the compilation. In the end, you'll have a "hello.node" file in your build/release folder.
\nTo actually use it in Node, just do the normal require, declare, and call:
\nvar myaddon = require("./libs/build/Release/hello.node");\nconsole.log( myaddon.hello() );\nWhat we get out of this is your console printing: "world". How does it actually get here from the init method? Well, this is where it gets a little hairy and abstract. THIS is where you get punished for learning C++ from Node.js. It all seems to be a weird glue between Javascript and C++.
\nRemember our init method:
\nvoid init(Handle<Object> target) {\n target->Set(String::NewSymbol("hello"),\n FunctionTemplate::New(Method)->GetFunction());\n}\nHere, we take the target, which is presumably the Node.js binding, and set a property on it - as if it were a dictionary or hash. Our hash key is "hello", and we use V8 to templatize and instantiate a new function that we've called "Method". There seems to be some V8/Node formality here, above and beyond C++ really - but THIS is how we've made our Node AddOn come to have the "hello()" method.
\nThere's a little more weird glue in the method itself:
\nHandle<Value> Method(const Arguments& args) {\n HandleScope scope;\n return scope.Close(String::New("world"));\n}\nI don't know how the scope is created here - it looks like something from nothing to me - but it looks like just by declaring scope, we've successfully created our Javascript/C++ glue - so when we close our scope we return our string of "world". It's still a little incomprehensible to me what's actually happening here - but hey....we know how to return a value to JS!
\n\nParameters also seem a bit weird. I wasn't able to explicitly say Method (int x, int y, etc), I think the parameters must come in as const Arguments& args....basically an array of arguments. The documentation shows us how to deal with type checking them and converting them:
\n if (args.Length() < 2) {\n ThrowException(Exception::TypeError(String::New("Wrong number of arguments")));\n return scope.Close(Undefined());\n }\n\n if (!args[0]->IsNumber() || !args[1]->IsNumber()) {\n ThrowException(Exception::TypeError(String::New("Wrong arguments")));\n return scope.Close(Undefined());\n }\n\n Local<Number> num = Number::New(args[0]->NumberValue() +\n args[1]->NumberValue());\n return scope.Close(num);\n}\nPay attention here how you ALWAYS are doing conversion. Whether IsNumber() or NumberValue(), you are basically extracting a value from a memory reference. This isn't Javascript anymore, where it will just type convert for you....this isn't even Java where it will just take your memory reference as a value. You need to RECOGNIZE that these are memory references, and you need to extract and convert explicitly to even get the integer value! And then to even get this number back to Javascript, you are throwing it into the Local
That's pretty much as far as I got, but I did create a more complicated example. Basically, my end goal here is to get x,y,z (3D) coordinates from C++ and pass to node. Since I don't have my fancy C++ library in place yet - I just make my AddOn generate random numbers and put them into a native JSON object and pass them on to Node.
\nHere's a gist of my "randomcoords.node" AddOn
\nAfter I build it, I can use it like so:
\nvar randCoords = require("./libs/build/Release/randomcoords.node");\nvar cursor = randCoords.getRandomCoords3D(600, 400, 100); // params are max values for random output\nconsole.log('{ "x":' + cursor.x + ', "y":' + cursor.y + ', "z":' + cursor.z + '}');\nBut of course - I don't log it, I send it on via a websocket to my browser to update the position of a sprite, but that's something for another day!
\n\nHope that all explained things - I know I can't explain everything, but I feel like I know just enough to be dangerous, and I hope that you can be dangerous too! I think this opens up a lot of power. I'm quite confident that you can do similar things in Python, Ruby, or whatever - but we're messing with the V8 engine here, and this can hopefully be knowledge transferred to Chromium someday so we can build the next Adobe Brackets or Spotify...it's quite exciting! My ultimate goal here is to use the OpenNI project within Node and transfer natural user interface data over to the browser via websockets. So far so good!
\n", "date_published": "2013-01-03T00:00:00Z" } , { "id": "https://benfarrell.com/blog/2013-01-02-sharkattack/", "url": "https://benfarrell.com/blog/2013-01-02-sharkattack/", "title": "SharkAttack", "content_html": "The SharkAttack (yeah!) started out as a radio show on Codebass Radio. It started out as an Adobe AIR based tool to CREATE the radio show. Since then, I've flipped the project around and turned it into a service that runs on Node.js. It still powers the radio show called SharkAttack, but it also powers my music discovery website at Blastanova.com.
\nThe SharkAttack service gathers up music from blogs all over the internet, downloads them, converts the videos to MP3, and packages them up as a playlist to play every Wednesday at 1pm EST on CodebassRadio.net.
\nIt's coming along pretty nicely, and I've done 50+ shows with it so far - and I'm always improving. Who knows, maybe someday I'll make it available for general use!
\nYou can find the SharkAttack Codebass Radio show page at sharkattack.blastanova.com
\n![\"shark\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2013/01/shark.png\")
RTE is a tech related talk show on Codebass Radio hosted by Adrian Pomilio, Dustin Graham, and yours truly. We take our (guessing here) around 40 years of combined development experience and talk about new tools, languages, tech, and more. Recent events, tech we're messing around with, comics, TV - whatever is relevant that day.
\nWe broadcast live on Codebass and Google+. You can catch us at runtimeexpectations.com
\n![\"rte_banner\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2013/01/rte_banner.jpg\")
Refresh the Triangle is the Raleigh/Durham/Chapel Hill chapter of Refreshing Cities. I co-organize the group with several awesome folks. We organize meetups that focus on web technologies and practices, covering design, development, project management, and others.
\nYou can find us on Meetup.com
\n![\"600_88807882\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2013/01/refresh.png\")
I work at a great little (getting bigger, so I don't know I can even say that anymore) startup called Digitalsmiths in Downtown Durham, NC at the American Tobacco Campus. Digitalsmiths is the industry leader in video search, discovery, and recommendations. We power many TV/movie experiences for studios, websites, and other emerging markets.
\nPersonally, I'm one of our elite UI developers who occasionally get turned on the server side. Given the spread of our market and clients, we get to work with Javascript, Flash, HTML5, Android, iOS, and more. It's a pretty nifty gig for those bold enough to take on whatever direction the market turns this week.
\nUpdate from 2024: Digitalsmiths was aquired by TiVo and the website isn't around anymore
\nCheck us out at Digitalsmiths.com
\n![\"digitalsmiths_home_wide\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2011/12/digitalsmiths_home_wide.png\")
Codebass Radio is "Where Geek and Music Combine". It's an awesome developer/geek/nerd oriented internet radio station founded by Vicky Ryder. I'm an eternally grateful to be her Senior Producer at the station - trying to step in and help where I can, as well as run two radio shows of my own.
\nCheck out CodebassRadio.net or listen live right now - There exists a show for every taste, and the schedule is chock full or fresh content every workday.
\n![\"codebassheaderlitenew\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2013/01/codebassheaderlitenew.png\")
Blastanova aims to be the new independent music discovery method. It strives for the middle ground between finding random bands online and listening to the top 40 on endless loops. To accomplish this, tons of great music blogs are scanned three times a day. Songs and videos are beamed up from the blogs to Blastanova’s playlists, Twitter, and Facebook streams every day.
\nBlastanova is my first serious attempt at Node.js work. The back end of this system uses Node.js, MongoDB, and some media utilities to go out and grab links to songs on various music blogs. From there I tweet them out and gather them up into a daily playlist powered with HTML5 audio. It works pretty well on various devices as a web app, but I've also made a Windows 8 Application from it. Coming soon will be Phonegap style Android and iOS apps.
\nCheck it out, especially if you'd like to discover fresh new music every day at blastanova.com
\n![\"blastanovalogo\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2013/01/blastanovalogo.png\")
How often do we get into "The Great OS Debate" with our colleagues and friends?
\n"Mac OSX has a superior UI to Windows, and I'll give you a point by point rundown why, along with diagrams".
\n"Windows is better for productivity, OSX feels like a Fisher Price toy"
\n"Android sucks, it's fragmented"
\nWhatever camp you're in, you seem to be passionate about it. As a tech savvy person, it's easy to pick a camp and argue about it viciously. Lately, I've been trying to accept all platforms equally - and while I certainly was guilty as an Apple hater before, I'd like to think I'm much more zen about things these days. I dig your platform, whether it be Windows, Apple, Android, whatever. All platforms have their good and bad points, and whether you can pick it up and use it to do the thing you're trying to do is the important thing.
\nAs I take a step back, its starting to remind me of the "Great Toilet Paper Debate". Which way do you hang your TP? Some viciously argue over, and others viciously argue under. Some will offer case by case arguments debating the merits of each. Some have blog battles over it. Some argue that over is more aestetically pleasing - supported by the fact that hotels go over to fold the edges and make it look even nicer. Others argue the physics - there will be less tear if under because of the gravitational force. Some argue that with the over style, it's easier for children and pets to grab it and make a mess. Others argue that under offers no wall clearance on some TP holders and will be less hygenic.
\nAll in all, I find the argument a bit silly, and I'm coming round to thinking the same for OS's. Certain factors can definitely make sense to way in on whether you go over or under - the pets/children option makes sense to me. But past that, it smacks of a silly battle - one that is GREATLY overthought. In the end, does it matter if you go over or under? To me, no - I can use it either way equally. Neither way in practice offers me any great benefit over the other. Any minor benefit offered, might be outweighed by other benefits - but to chart and plot and diagram why seems a wasted effort when you could just go with the flow and not pick a silly battle like that.
\n", "date_published": "2012-12-17T00:00:00Z" } , { "id": "https://benfarrell.com/blog/2012-12-12-starting-out-with-openni/", "url": "https://benfarrell.com/blog/2012-12-12-starting-out-with-openni/", "title": "Starting out with OpenNI", "content_html": "I've been digging on the Kinect for a little while now. I first started toying around with it last year, and gave a presentation at NCDevCon on using it with the web stack. Here's my write-up and slides from it
\nIt's been a few months - and unfortunately I haven't seen anything from Kinesis.io. I was really pulling for them to pull off this AMAZING Kinect/Web integration and sell it. I would buy it like 10x over! I ended up doing demos using KinectJS.
\nNow, KinectJS is pretty amazing itself, but it has a few dependencies that I'm not so much OK with. It requires Adobe AIR - which is almost OK. I used to do some AIR development and it's a pretty slick tech. Unfortunately, they stopped support for Linux, which I tend to use now if I'm working on web pages. It also requires Windows. See previous comment. It also uses the Microsoft Kinect SDK, which while amazing, again...requires Windows.
\nMy problems basically boil down to several layers of tech, of which NONE are cross-platform. So there's really just no hope there for anything to come up my way, really. I don't mind Windows, I really don't. I even have Windows 8, and despite what they tell you, it's a nice OS. I'd just prefer not to be locked in, especially with web technology, which I REALLLLLLY want to use with the Kinect and show all my friends who use Mac or Windows.
\nAnyway, that's where OpenNI comes in. It's LIKE the Kinect SDK, but it's cross platform. OpenNI is actually a non-profit founded by a few companies including PrimeSense. Primesense is the original inventor of the Kinect and SDK which later got sold to Microsoft.
\nMain takeaway - I'm RUNNING IT RIGHT NOW ON UBUNTU! Can't you just feel my excitement? You can see OSX, Ubuntu, and Windows downloads all on OpenNI's download page. I initially had some trouble getting Linux to work. I was using the Mint distribution before, and it looked like some of the graphics libraries weren't quite up to par with what OpenNI needed.
\nTo get everything working, I followed the instructions from this webpage on Ubuntu. Oddly, they do mention it works in Mint...guess they are just smarter than I, or more likely, I just didn't find these great instructions before I decided to switch to Ubuntu (probably both). Main takeaway is to use the "Unstable" downloads of the OpenNI binaries, the sensor drivers, and NITE. This applies to you Mac folks as well - you can't even access the downloads unless you choose "Unstable" - don't worry, it seems to work great. The main OpenNI binaries are like the heart of OpenNI, NITE is the middleware - the brain that reads and interprets the data, and the sensor drivers....well they are hardware drivers.
\nOnce you've installed all of this stuff, go on in to the OpenNI->Samples->Bin-x64-Release and crack open "NiViewer". You'll see a yellow depth view to one side and a RGB camera view on the other side and some info down below.
\n![\"sampleviewer\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2012/12/sampleviewer.png\")
\nNI Viewer Application
Now, hopefully you have Java installed or this won't work. Double points if you have the JDK (that's Java Developers Kit) installed instead of the JRE (Java Runtime Engine). Because, while this will work on the JRE, we can't hack/code our own without the JDK.
\nBut we also need an IDE! Or at least I do. I barely know Java, so I'm not going to be editing with a text editor and compiling with command line utilities. A lot of folks will go for Eclipse. I don't blame you, it's popular. I've been actually digging JetBrains Intellij Community Edition (community edition is free!)
\nIt was actually pretty simple from here on in, once I worked out the quirks. In Intellij, I imported the entire OpenNI->Samples folder as an existing project. Originally, I had tried importing each sample individually, but there were a fair amount of syntax errors and unknown references from messed up paths - so importing the whole thing was way easier.
\nThe first experiment I tried was running SimpleViewer.java. If you open to Samples->SimpleViewer.java->org->OpenNI->Samples->SimpleViewer, make the SimpleViewerApplication.java run. You may get a runtime error - I did. Mine was caused by the "node configuration" file. Basically, this configuration file tells you which "Production Nodes" are running. Right now, I just understand that there is a depth camera node and an RGB camera node. There are probably much more.
\nAnyway, the SimpleViewer.java class has a private var (Line 45) which says that the config is here: "../../Config/SamplesConfig.xml". For me, it wasn't - it was at "Config/SamplesConfig.xml". So that fixed my issue, and then running the sample produced a nice, fancy depth mapped video feed.
\n![\"depth\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2012/12/depth.png\")
So, I thought that was pretty cool! That's just the depth camera - there's also the RGB camera on the Kinect. Both cameras are accessible PRETTY much the same way. The depth camera is driven by the "DepthGenerator" object, while the RGB camera is driven by the ImageGenerator object. Then, there is a whole MESS of issues which involve reading a different number of bytes per pixel if you're posting a greyscale pixel or a RGB pixel - hint....each color is another 8 bytes.
\nIn the end, getting the color image to work involved some research and stealing from the web... again I'm not a Java expert. I'll post a final Gist after I do some more magic, but here's my RGB video feed working:
\n![\"rgb\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2012/12/rgb.png\")
Now, the depth camera is mildly interesting. It's not something you see everyday. And despite the RGB camera being tricky because of bytes and such things, it's definitely a "meh".
\nWhat if though, I could combine the depth sensing data WITH the RGB live video feed???? Awesomeness, I tell you!
\nSo I went in to the spot where I create each pixel, and said - "Hey you pixel, if the depth data in the EXACT SAME spot is greater than 1600, then you will just be red." Basically here's my block of code:
\n while(depth.remaining() > 0)\n {\n int pos = depth.position();\n short pixel = depth.get();\n if (pixel > 1600) {\n pixel = 0;\n img.get(); img.get(); img.get();\n clrbytes[pos*3 + 0] = 127;\n clrbytes[pos*3 + 1] = 0;\n clrbytes[pos*3 + 2] = 0;\n } else {\n clrbytes[pos*3 + 0] = img.get();\n clrbytes[pos*3 + 1] = img.get();\n clrbytes[pos*3 + 2] = img.get();\n }\n }\nSo this leads to a situation where anything close enough up to the camera has full RGB pixels. Anything far enough away is reduced to a red, flat background - like so:
\n![\"backgroundreplace\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2012/12/backgroundreplace.png\")
Obviously not perfect, but hey I'm just learning and feeling it out. One MAJOR problem I had early on was that my outline was bigger and kinda distorted from the shape of my body. It turned out to be a pretty obvious thing that you'd never think of. The problem is that both the RGB camera and the depth camera are in two different physical locations on the Kinect hardware. So you have 2 completely different perspectives of the image you're trying to capture.
\nI'm going to use a word at the start of the next sentence, and I'm not exaggerating. MAGICALLY, you can transpose the depth data to the perspective of your RGB camera. If you think about it, the depth data is an array of geo,metric points. So OpenNI (or the Kinect, I'm not sure) can shift those mathematically to the perspective of the RGB camera. It, of course, doesn't work the other way around (transposing RGB pixels to the perspective of the depth data). It's all done simply by saying:
\ndepthGen = DepthGenerator.create(context); // initial creation of depth camera\nimageGen = ImageGenerator.create(context); // initial creation of RGB camera\ndepthGen.getAlternativeViewpointCapability().setViewpoint(imageGen); // transposing the view\nSo, once I got all that working, I made my first art installation dubbed "Ghost Hands":
\n![\"ghosthands\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2012/12/ghosthands.png\")
Not perfect, but we're just having fun and seeing what we can do here. But again, this was just accomplished by restricting how far the camera can see depth wise and replacing what's too far away with red pixels. I step out of range, and put my hands in range.
\nWe're BARELY scratching the surface of what the Kinect and OpenNI can do here. It can do MUCH cooler things, I promise you. Gestures, skeletal tracking, better background removal (I would assume), and more. I'm just getting started with my rudimentary experiments and lack of formal Java knowledge.
\nPoint is - this is really a lot of fun, and now that we've covered the simple stuff, it'll hopefully fly from here!
\nHere's a Gist of the final hacked "SimpleViewer.java" class of the SimpleViewer application. Please note that I renamed MY class to SimpleViewerCustom so I could keep the original pristine SimpleViewer.java class. So be sure to note that and change your class constructor to match!
\n", "date_published": "2012-12-12T00:00:00Z" } , { "id": "https://benfarrell.com/blog/2012-11-27-hey-idiot-check-out-d3-js/", "url": "https://benfarrell.com/blog/2012-11-27-hey-idiot-check-out-d3-js/", "title": "Hey Idiot, Check Out D3.js", "content_html": "Sorry, of course I don't really mean you're an idiot. I personally was always the type to turn down the "For Dummies" books when I was learning web technology years and years ago. My thinking was something along the lines that I am:
\n![\"tumblr_mdfhbwDAxb1qd7xzbo1_500\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2012/11/tumblr_mdfhbwDAxb1qd7xzbo1_500.gif\")
Hey Jerk, I'm not a dummy!
\nBut then came along my immediate and burning need for a bar graph on my webpage. It was then I saw the light. Nowwwwwwww, I WANT to be a dummy. Why? Bar graphs are god awful boring. I didn't want to do it. I had a need to visualize some data, but the prospect of actually doing it sounded DUUUULLLLLLL.
\n![\"cats14\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2012/11/cats14.jpg\")
Initially, I tried to get some free slave labor, but the only warm bodies in the room were either visually handicapped or just too lazy.
\nSo I went over my options
\nIndeed I did hear about D3. I heard about it because while I was giving my Javascript Powered Kinect presentation at NCDevCon, a very smart gentleman in another room named Ben Clinkenbeard was giving a presentation on it.
\nNow, while I wasn't able to see the presentation, that was enough for me. I figured I'd give it a shot. But I had a severely bad attitude going in. "Grumble grumble bar graph"....."grumble grumble every other JS framework I've tried takes at least a few days to get up to speed with with".
\nOK fine....clear my mind, "I'm a dummy.....I'm a dummy. Copy a bunch of code from the samples and move on."
\nI found a decent sample after a short time looking: http://www.recursion.org/d3-for-mere-mortals/ I copied the bar graph code. And found it INCREDIBLY simple to run with it. It was easy to read and understand.
\nFirst, we'd need to create the d3 instance:
\nvar svg = d3.select("body").append("svg").attr("width", w).attr("height", h).attr("id", "myid");\nThen, each element in the bar graph had a block of code with it. Like the actual bar itself:
\nsvg.selectAll("rect").data(dataset).enter().append("rect")\n .attr("x", function(d, i) {\n return i * (w / dataset.length);\n })\n .attr("y", function(d) {\n return h - (d * 4) - 20;\n })\n .attr("width", w / dataset.length - barPadding)\n .attr("height", function(d) {\n return d * 4;\n })\n .attr("fill", function(d) {\n return "rgb(0, 0, 255)";\n });\nYou can see here that we're creating (appending) a rectangle with a specific color at the specific x-axis position of the data (dataset). The height is the actual value of the data at the x-axis index.
\nOur copied example also put a numeric display of the count about the bar:
\n svg.selectAll("counttext")\n .data(dataset)\n .enter()\n .append("text")\n .text(function(d) {\n return d;\n })\n .attr("text-anchor", "middle")\n .attr("x", function(d, i) {\n return i * (w / dataset.length) + (w / dataset.length - barPadding) / 2;\n })\n .attr("y", function(d) {\n return h - (d * 4) - 44;\n })\n .attr("font-family", "sans-serif")\n .attr("font-size", "12px")\n .attr("fill", "black");\nI then thought that, maybe I should put a date label below each bar to indicate how many hits on that day (the nature of what I wanted to show).
\n svg.selectAll("labeltext")\n .data(labels)\n .enter()\n .append("text")\n .text(function(d) {\n return d;\n })\n .attr("text-anchor", "middle")\n .attr("x", function(d, i) {\n return i * (w / dataset.length) + (w / dataset.length - barPadding) / 2;\n })\n .attr("y", function(d) {\n return h;\n })\n .attr("font-family", "sans-serif")\n .attr("font-size", "11px")\n .attr("fill", "black");\nYou'll notice that to stick this text at the base along the x-axis - I make it set the "y" attribute to "h", the height of the entire display. And above, I made the "y" value of each bar "h-20", to give a 20 pixel buffer for text to show.
\nI added even more stuff too! The final result stacks a count of video hits and audio hits on a single bar, and displays each day for the past 30 days how many of each (and total hits) there are.
\nI also stacked the counts below the x-axis label, so I padded the bottom of the graph even more than my simple examples above.
\nYou can see my results here: http://blastanova.com/play-trunk/dailyreports.html
\nProbably the hardest part was organizing the JSON data I wanted to drive the graph and slicing it a few different ways. As far as I could tell, each of your bar graph attributes need a one-dimensional array driving the attribute. it makes sense, but my situation in particular was a bit more complicated!
\nNo problem though, it all worked out. And I got a bar graph and an education in D3.js in a night!
\nOn a side note, it uses SVG (scalable vector graphics)! How cool is that! I noticed some blurriness on the sides of a few bars as it tried to anti-alias, but that's probably solvable (and maybe I shouldn't have overlaid my bars when I could've stacked them).
\nSo right! My suggestion to you? Use what I write to inspire you to try out D3 and copy from better examples than mine. In the meantime, bar graphs don't suck anymore....I'm kinda interested to know more!
\n", "date_published": "2012-11-27T00:00:00Z" } , { "id": "https://benfarrell.com/blog/2012-11-16-where-the-are-my-windows-8-ui-button-graphics/", "url": "https://benfarrell.com/blog/2012-11-16-where-the-are-my-windows-8-ui-button-graphics/", "title": "Where the *#$%! are my Windows 8 UI Button Graphics?", "content_html": "THIS is the question I was asking myself. I was in the middle of porting my web app over to Windows 8, and said to myself - "Self, this looks good, but I think it could be a little more metro-ish".
\nOf course Metro isn't the term anymore, now it's Windows 8 UI - but the point is the same: Windows 8 has a pretty distinctive look, and if you're going to develop apps for it, you might want to follow suit. Do as the Romans do....in Romantown....so to speak.
\nWindows 8 makes this fairly easy for you - from the VS Express home screen, you can elect to do a blank app or a grid app or several others. It will auto-populate a fair amount of templating into your brand new project to make it look all Win8-ish. I've noticed a few areas of the basic template that makes the look Win8's own.
\nAnimation
\nIf you elect to use the page navigation model, pages will slide in. Its a nice subtle navigation effect. It's also completely exposed in Javascript, so you can muck with it if you wish. It's all in the navigation.js file that Visual Studio copies over for you. It's also cool that there is a specific hook that you can include on a page definition. If you define the function getAnimationElements on your page JS like so:
\nWinJS.UI.Pages.define("/pages/about/about.html", {\n ready: function (element, options) {\n },\n\n getAnimationElements: function() {\n return myelement;\n }\n...it will let you define which DOM element you'd like to animate. You can pass back null as well if you don't want animation on that page. I never did figure out why my animations for pages were freaking out and stuttering when I messed with things heavily on the HTML side, but by blindly putting things back in the code, I see that the section element seems pretty important here:
\nSo leave that in unless you've cracked the mystery and can explain what happens better than I.
\n\nFont
\nNext up is the choice of font. MS seems to LOOOOOOOVE their own font Segoe UI. So, you should probably use that, especially on big headers and menu navigation groups and grids. They make it pretty easy to use as well. I've always used explicit ems or pixels in my CSS properties, but you can see values like "x-large", "xx-large", "medium", and "small" all over the place. I didn't know this was actually valid in CSS, but briefly looking it up, I see that it's not just limited to Win8!
\nFor things in an <h1> tag, this will be the standard header that most Win8 apps will use, so specifying any sort of font here is unnecessary. Also cool is the "win-type-ellipsis" class that you can put on elements. This class will truncate any text that's too wide and add a "..." to the end. Very cool, especially in a fluid layout design when you can drop into snapped mode and things are very narrow.
Buttons
\nLastly are the #$%#$% buttons I couldn't find. Or at least it took me like an hour to figure out what was going on. When doing page navigation on a typical page navigation demo app, they have a nice looking back button. If you traverse to a page, click the back button that appears to go back in history.
\n![\"backbutton\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2012/11/backbutton.jpg\")
To me, this back button, the spacing, and the fonts around it just SCREAMS Windows Modern UI. So if you want your application to look like a Windows 8 app, you should probably not fight the templates and use something similar.
Making your very own back button is easy, simply do this:
\n<button aria-label="Back" type="button"></button>
I've seen other similar looking buttons on media player controls in Windows as well, so it's best to duplicate that look on your buttons as well if you can.
\nHow do you do that? Well, it was a pain to track down. I was on the wrong path looking for png image references in the WinJS source CSS files. I even scanned through the Javascript trying to find some reference to how this worked!
\nThe images are nowhere to be found, and that's because images aren't used! Instead, WinJS uses the Segoe UI Symbol font. It's like Wingdings except it looks better. This font by MS is full of little icons. Through the magic of CSS and border radii, a thin circle is drawn around the DOM element. The contents of that element are the Segoe UI Symbol character itself.
\nNot typically doing anything funky with CSS on a normal basis myself, I was surprised to see the following CSS selector and definition:
\nmybutton::before {\n\n _content: \"\\\\E102\";\n\n}That backslash is an escape character that lets you escape from the traditional tedium of entering real text and lets you enter unicode MADNESS. What you'd do is load up the Windows Character Map program, select the Segoe UI Font, and select the character you want, and copy the code for it. Here's me checking out the code for an alarm clock:
\n![\"charactermap\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2012/11/charactermap.jpg\")
So here, I'd put backslash 23F0 into my CSS to make an alarm clock. My alarm clock CSS button might look a little something like this:
\n.alarm {\n display: inline-block;\n min-width: 0;\n min-height: 0;\n background-clip: border-box;\n box-sizing: border-box;\n border-radius: 50%;\n border-width: 2px;\n border-style: solid;\n padding: 0;\n padding-top: 3px;\n margin-top:10px;\n margin-bottom:10px;\n text-align: center;\n width: 60px;\n height: 60px;\n font-size: 21pt;\n}\n.alarm:hover, .alarm:hover:active {\n border-width: 2px;\n border-style: solid;\n border-radius: 50%;\n background-color: rgba(255, 255, 255, 0.2);\n}\n.alarm.play::before {\n font-family: "Segoe UI Symbol";\n font-weight: normal;\n font-size: 24pt;\n content: "\\E102";\n vertical-align: middle;\n}\nAnd so, there you go - a few different Window 8 UI type touches that you can use on your application! Including the one that inspired this post, that I swore under my breath for a good hour while trying to find.
\nThere's a lot of Windows 8 hate out on my twitter stream. Some folks might have problems with its useability, and that's fine. Some people think the Surface RT will fail, and that's fine too. I don't agree with any of that, but it certainly is subject to opinion and anyone's guess what the future will hold.
\nI do want to tell you one fact, and that is that Windows 8 development is a smooth ride. After doing Flash, Android, iOS, web, and more - Windows 8 is my new favorite. Maybe I like it because it's brand new and something shiny, but it's definitely a solid, thought out environment for making apps (or rather applications as MS would say). Hopefully the ecosystem will grow and grow and compel me to make Windows 8 app after Windows 8 app. But that's anyone's guess if that will truly happen. Android successfully competes with iOS, but on the other side of things....well there's RIM. Windows 8 will probably fall somewhere in that wide spectrum.
\nGiven the tenuous nature of whether an ecosystem fails or succeeds, MS has to give us some hook to get us developing apps. What makes it worth our while? Well, first of all, it's Windows. Windows is big....so just by the nature of them being the huge gorilla in the room, they will probably succeed to some degree that makes things worthwhile to developers in the short term.
\nThe OTHER hook is that in addition to XAML/C# that the old Microsoft salts are used to, we also have the ability to develop our applications with HTML, Javascript, and CSS. This is huge! I have a music based web app up at blastanova.com and was able to get the thing running in Windows 8 in a matter of hours. Yes, its a couple weeks later and I'm still being fussy making it a fully fledge "metro" looking app - but the guts were done in a matter of hours by recycling my existing HTML/CSS/JS.
\nI think this is a big deal. MS has given us a platform that may or may not be successful, but making applications for it is a pretty painless process. There were some roadblocks though! And given how common jQuery and Google Analytics are on our web apps today, its something that we'll all have to work through...so let's get started!
\nI'm ashamed to say that I don't use jQuery for much besides DOM manipulation/selection and loading data. Yah, I could probably do this without jQuery, but I find myself finishing a project and jQuery is still there. I throw up my shoulders and say...."ehhhh, I'd rather not rip this out". Anyway jQuery works pretty well for this stuff, with just a one problem.
\nSetting innerHTML fails in Windows 8. Getting the innerHTML of an element is fine, though. The theory is that if you inject random HTML into a div, you're doing something unsafe and you better MEAN it. That's why MS makes you do the following:
\nWinJS.Utilities.setInnerHTMLUnsafe(myelement, my content);
\nSo, since you're using this WinJS specific thing, Microsoft says to itself...."You know, this guy actually means it. If he's doing unsafe application development here, it's on him....but we tried to stop it." . I do believe that there is another utility to set the inner HTML, but it doesn't work on dynamically loaded data, so I haven't had the pleasure since most of my work involves loading external JSON data and whatnot.
\nAnyway, we use the new utility function, but Visual Studio STILL complains at us! Don't get me wrong, your application still runs, but there is a warning issued:
\nHTML1701: Unable to add dynamic content ' <link/><table></table><a href='/a' style='top:1px;float:left;opacity:.55;'>a</a><input type='checkbox'/>'. A script attempted to inject dynamic content, or elements previously modified dynamically, that might be unsafe. For example, using the innerHTML property to add script or malformed HTML will generate this exception. Use the toStaticHTML method to filter dynamic content, or explicitly create elements and attributes with a method such as createElement. For more information, see http://go.microsoft.com/fwlink/?LinkID=247104\nWhy the warning if we don't use innerHTML? Well, turns out that if you use jQuery, you are calling it simply by embedding the script. Deep in the bowels of jQuery, we see this line:
\ndiv.innerHTML = " a";
jQuery just runs this when you embed it. So, what I've done is grabbed the unminified source, and commented out the line and all is well. In jQuery 1.7.1, its at around line 1352. I then saved this unminified source as my special modified version of jQuery and included it in my project.
\nI had other troubles with jQuery 1.8 - and I never did look into it properly. 1.7 seemed the easiest way to go.
\nKeep in mind, that there are other ways of updating your DOM that aren't cool either. jQuery's "append" function uses the innerHTML function under the hood so it's not something that's available to you anymore. And that's actually just a little annoying, since there doesn't seem to be an equivalent appendUnsafe function in WinJS, so you have to read out the element, do a string append or something similar, and reset the innerHTML using WinJS's utility function.
\nAll in all, that was my only issue with jQuery - maybe there are others, but I didn't encounter them. It DOES make sense to sandbox this unsafe behavior, so I don't think you can fault MS for this one really.
\n\nSurprise, surprise, another security issue. So, here's the deal - under normal conditions, you can include JS files in your DOM however you wish. This isn't a surpise....it just means you deal with Javascript the normal ways that you would on the web. Include in your HTML, attach programmatically, whatever.
\nThe unfortunate bit is that you can't load an external Javascript file from the web into your project. Only local JS files built into your application may be run. This is problematic, because that's the only documented way to load Google Analytics. To use GA, you include the script http://www.google-analytics.com/ga.js on your page.
\nThere IS a workaround, but it's not incredibly pretty. Ok, it sucks - but it works. The WinJS security model is such that you CAN load external scripts if you do it in an iframe. The iframe is outside the security sandbox, and in theory can't wreak havoc on your application when stuff loads in.
\nYou might be saying, OK smart guy - I'm gonna make an HTML page in my project that hosts Google Analytics, and then have that as an iframe in my application. Sorry, that doesn't work either. It turns out that local iframes aren't allowed - they MUST be fetched online with the security sandbox model they have set up.
\nDAMMMIT. OK, this is where the workaround starts sucking. I made my iframe page, put my Google Analytics integration on it, but then hosted it on my website. So again, to actually use GA, you need to host a remote webpage as an iframe within your locally run application. But, once you got that in your head, it's pretty straightforward.
\nHere's my remote file's Javascript (iframe source):
\n console.log("iFrame GA Initialized");\n var _gaq = _gaq || [];\n var ga = document.createElement('script'); ga.type = 'text/javascript';\n ga.async = true;\n ga.src = ('https:' == document.location.protocol ? 'https://ssl' : 'http://www') + '.google-analytics.com/ga.js';\n var s = document.getElementsByTagName('script')[0]; s.parentNode.insertBefore(ga, s);\n\n function track(event) {\n console.log("iFrame GA Event: " + event.data.join(" | "));\n _gaq.push(event.data);\n }\n\n window.addEventListener('message', track);\nSo, the HTML file is completely empty, except I put this script in a script tag in the head element of the page. Now, those familiar with Google Analytics know that there are a few methods to track. I've chosen the "_gaq" method especially because it works well with custom events. In my application, I can do all the setup, initialization work, pageviews, and tracking. All this script does is take incoming iframe messages, and posts the data to the "track" method. The data then gets "pushed" onto the _gaq object.
\nAgain, we don't care about what that data is - the script is pretty dumb and doesn't know about our account, setup, nothing. In our application on the other hand, we run "postMessage" on our iframe to actually run the nitty gritty of the GA implementation.
\nIn my iframe, used as:
\n ...\n...I use the following Javascript (after of course the "onGoogleAnalyticsInitialized" callback handler is fired):
\n document.frames['googleTracker'].postMessage(['_setDetectFlash', false], "http://blastanova.com");\n document.frames['googleTracker'].postMessage(['_setAccount', self.analyticsID], "http://blastanova.com");\n document.frames['googleTracker'].postMessage(['_setDomainName', 'none'], "http://blastanova.com");\n document.frames['googleTracker'].postMessage(['_trackPageview', "/" + self.appName + "/home"], "http://blastanova.com");\nNow, a couple important things to call out here...
\nThe "http://blastanova.com" is not something that's important to our Google Analytics implementation, but it IS imperative to our iframe implementation. This needs to be an http(s) URL or the iframe postMessage will not follow through. And no, your WinJS document.location will not work here because it's not an http(s) URL. You will get a DOM Error 12 or something similar.
\nSecondly, you've probably never heard of "_setDetectFlash". This doesn't seem to be widely documented. But if we take this out, we will get a warning from Visual Studio that we've tried to create an ActiveX object and it was blocked. The application still seems to run fine, but I wanted to get to the bottom of this.
\nIt turns out, that FOR SOME UNKNOWN REASON, Google Analytics puts a Flash embed on your page. Who knows why. The important thing is that Google gives you the power to turn this off. Do this as the first thing when you interact with your iframe.
\nOnce this stuff is in place, all the Google Analytics calls work fine. Go ahead use "_trackEvent", everything will be OK!
\n\nThat's OK, me neither. There are actually other ways to do Google Analytics on Win8. Unfortunately, they seem to all work only for XAML/C#. It looks like right now, WinJS is sort of a second class citizen, with the packaged DLL's seeming to come out for XAML/C# as the first priority. It takes library developers a little extra effort to package this stuff up for WinJS, and sometimes they just haven't done it yet.
\nThis solution on MS's open source hub, for example, seems like a perfectly acceptable non-hacky solution. But, I just couldn't see anything about getting it working on WinJS, so I assume they haven't gotten around to wrapping it that way yet.
\n\nAnyway, hopefully now you can get these major tools (jQuery and Google Analytics) working for you instead of against you in your Windows 8 application. Good luck!
\n", "date_published": "2012-11-09T00:00:00Z" } , { "id": "https://benfarrell.com/blog/2012-10-24-did-html5-and-mobile-kill-the-volume-slider/", "url": "https://benfarrell.com/blog/2012-10-24-did-html5-and-mobile-kill-the-volume-slider/", "title": "Did HTML5 and Mobile Kill the Volume Slider?", "content_html": "Seems so simple and obvious doesn't it? If you make a media player, whether audio or video, you should include a volume slider.....right?
\nNow that I'm developing web apps that extend onto the mobile screen, I got the feeling that a volume slider wasn't needed. I couldn't quite articulate why - but it was along the lines of the fact that you have a physical volume switch in your hands already as you hold the device, so you get tactile control. I also had in the back of my mind that I wasn't seeing so many volume sliders anymore, especially on mobile - but couldn't back that up with actual facts.
\nBut, as I'm working on my daily music playlist web app at http://play.blastanova.com, I do like to get user feedback. I took an informal twitter poll asking people if they think volume sliders in their media players are necessary, or if they clutter things up too much.
\nTwo people responded, both saying that volume sliders are better than no volume sliders. So OK, I know when I'm beat! It wasn't my opinion, but I would certainly give it a shot and see how it worked. I could always throw them away if it's horrible!
\nSo, I set off to work. Interestingly enough, I wasn't quite sure how to begin. I had a few options that I knew of. The first was to use a plugin like jQuery UI. Problem is, I was only using jQuery - not the bloat that is jQuery UI yet. To be fair, it was only bloat because the proposition is to include a ton of extra crap in my code JUST to get a slider to work. Luckily I can pare down what I need in my custom jQuery UI library through the builder. That's just what I did - for a time. I got a slider working, even custom styled it chopping away block after block of unnecessary CSS from my style. Eventually I got something a little bloated, but not horribly so.
\nThen I thought to myself - "Damn, you're basically doubling the size of your project to include a UI element that you aren't even sure you need!". OK, lets think second option - writing a slider from scratch....ummmmm....no. While I do enjoy UI challenges, I don't enjoy creating commonplace UI components from scratch. If I needed to invest the time and effort, that would be one thing, I'd suck it up and dive in.
\nLast option - the range slider! OK I actually forgot about this one. As far as I'm concerned, the only new HTML5 elements worth talking about are the canvas, video, and audio tags. So, its easy to forget something this unsexy. Fine though, let's look into this and see how it works.
\nIt turned out quite easy to pop in - though I did want it to be vertical. After Googling for some time, I could see no other option than doing a CSS rotation transform on the slider. Using CSS3 seems pretty overkill and weird for this. I had some trouble positioning it with the rotation rules applied, but eventually got it where I wanted it. Of course, after i did that, I decided a horizontal slider in a different spot on my page would work better visually after all.
\nLast stop was styling the slider. Eventually I got something with a round thumbnail with a trough that looked something like this:
\n![\"volume\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2012/10/volume.jpg\")
The HTML was pretty simple:
\n<input id="volume" type="range" value="80">\nAnd the CSS used a webkit vendor prefix:
\ninput::-webkit-slider-thumb {\n\n -webkit-appearance:none;\n\n width:30px;\n\n height:30px;\n\n -webkit-border-radius:30px;\n\n border-style: solid;\n\n border-color: #1a1a1a;\n\n border-width: 1px;\n\n background-image: url(\"../images/vol.png\");\n\n}Fine, now how about Firefox? Does that work? Lets test in that.... BOOM! Nope, doesn't work. Well, that's PROBABLY because I have a Webkit vendor prefix on the thumb and it's just breaking down entirely. OK, to Google in search of customizing our range slider for Firefox!
\nWell, after some searching, not only does Firefox not support range slider customization, but it doesn't support the stupid HTML5 range slider at ALL. Great, well, I did all this work...let's just add in the volume slider, hide it in Firefox, and see how I live with it on the page. Let's see if I like it at least in Chrome.
\nI sat with it for a few days. I never really found myself reaching for it. Usually, I'd just reach for my computer volume (hardware or software) to turn my entire system down. But eventually, I was on a mobile device, and wanted to test it out and see if it worked.
\nSo....the slider worked, but it had NO EFFECT ON THE VOLUME!!! I swear, every time I turn around, something annoying pops up with this. I went full on "TAKE COMMAND MODE", and went and installed the Android development tools on my system for the first time. I rigged up Chrome debugging over USB so I could view my console log, DOM tree, and all the Chrome goodies I could get right on my desktop. I expected some red text in my console indicating a Javascript error. NOPE, completely silent.
\nFine...let's Google "HTML5" and "set volume mobile" (which I should have done in the first place).
\nTurns out....well this.
\nBoth iOS and Android's version of HTML5 don't support setting the volume on video or audio elements. The iOS page specifies that you may use the hardware switch to decrease and increase the audio of the system.
\n.....And we're back to square one. It would seem as if I had the right inclination all along on mobile. It's expected that users will adjust audio physically on web pages. Of course there is the unfortunate side-effect that you are controlling all applications on your device at once.
\nNow, if I am making a similar experience on the desktop, do I include audio adjustment? Well, it might be a good idea to match the user experience from one to the other. So a easily recognized feature like audio that is usually placed next to your main playback controls is hard to miss when it's there on some experiences but not others. It would be one thing if we just NEEDED TO HAVE IT. But we don't. As a feature that is on the chopping block already, keeping a consistent user interface is a good reason to leave it out on the desktop.
\nAnd then there's the ease of implementation. Holy crap, it's like you're either doubling your project size with jQuery UI or taking on a second job to do it from scratch. Or you know, just make it work on webkit only.
\nAfter all this, you start to wonder if it's really that important - and i'm beginning to think it's just not worth it....to anyone. Maybe I should just kill it from my project. I might imagine folks are doing similar things and finding it's better to leave it out. So is it dead yet? Did workflow and capabilities kill the volume slider on the web?
\nAs a last note...the slider seems alive and well in native apps. I'm watching Netflix right now on my Galaxy Tab 10.1. I typically leave it charging when I'm watching Netflix. So, while it's in landscape mode and charging (with the charger sticking out the top), the device itself actually rests on the side where the physical power button and volume are. As such, when I watch Netflix, the physical volume is completely inaccessible. I'm not sure about the iPad, as I don't have a fancy case for it yet.
\nAnd wouldn't you know it - Netflix has a nice little horizontal volume slider :)
\n", "date_published": "2012-10-24T00:00:00Z" } , { "id": "https://benfarrell.com/blog/2012-10-05-javascript-youre-beautiful-why-so-defensive/", "url": "https://benfarrell.com/blog/2012-10-05-javascript-youre-beautiful-why-so-defensive/", "title": "Javascript, You're Beautiful. Why so Defensive?", "content_html": "Look, I'm sorry, but Javascript is a weird language. There's nothing wrong with that, certainly, but it seems Javascript developers take this factopinion as a slap in the face. They get all defensive.
\n"There's nothing weird or strange about my language-baby, you're just stupid because you can't learn it" they seem to say.
\nPopular negative opinions on Javascript range from the lack of strict typing on variables, lack of classical object-oriented structuring like Classes, Interfaces, Inheritance, etc. This is all stuff that a lot of the more traditional languages have had and loved for years. Developers in these languages get their groove on with these tools, and sometimes can't imagine working without them.
\nThink of it....like a highschool. You've got your popular kids (the Java developers), the athletes (the C++ developers), and the rich kids (.NET C# developers). They all talk about popular kid things - sports, girls/boys, cars, makeup, and more sports.
\nNow, I don't know about you but I didn't sit at the cool kids lunch table in school. Yes....my friends were awesome, and we were cool on our own, but we weren't the POPULAR kids. Some of us like sports, cars, and some of us wore makeup - but we didn't talk about it all the time. We talked about how we couldn't wait until Saturday morning to watch the next episode of "The Tick" cartoon, we talked about guitars, bands - we acted out sketches from MTV's "The State". We ran our schools public access station. We had a blast, and we loved it.
\nLooking back, maybe we were a little freaky - but we owned it. I don't know that we really had that much in common with the popular kids. I still to this day can't identify a car if I'm sitting in it, and I sure as hell don't know sports, but I'm doing my own thing - and I know it.
\nThat's exactly like Javascript developers. They might be a little freaky, but they're no less beautiful than the popular kids. And you know, the popular kids might look down on them because when they try to strike up a conversation interfaces, strict typing, etc - they just aren't on the same page.
\nHere's the difference though - some Javascript developers seem to be saying "Cars are for weaklings, try riding a bike next time". "Sports? Pish-tosh, that's for you common folk un-intellectual folk. I'm better than sports, surely you are as well"
\nNo - you Javascript developers know who you are . You're doing it wrong. Javascript is different. Javascript is a freak, but Javascript is beautiful too - and yes you might not need all the frills in the other langauges, but don't act like others are stupid for asking about them. Calling somebody stupid and acting like you're better than them is a barrier to the world having a good opinion of Javascript.
\nI know, I know - sometimes you'll get made fun of by the popular kids. Its OK, it happens - but if you dish it back, nobody wins. Just try to educate, and they are too closed minded to listen, ignore.
\nAnd it goes both ways - listen to them too. The stuff they know is popular for a reason. Coffeescript, Dart, and Typescript are there for a reason. Be OK with it and understand it, even if you don't prefer it.
\nDon't be negative, spread a positive message: "Javascript is just a little freaky, but it's OK... it's beautiful and you'll get used to it."
\n", "date_published": "2012-10-05T00:00:00Z" } , { "id": "https://benfarrell.com/blog/2012-10-03-morph-your-media-with-node-js-redux-spawning-proccesses/", "url": "https://benfarrell.com/blog/2012-10-03-morph-your-media-with-node-js-redux-spawning-proccesses/", "title": "Morph your Media with Node.js Redux - Spawning Processes", "content_html": "Sometime in September, the music discovery service I run stopped discovering YouTube music. It was OK, because the playlist I offer is MP3 only - but I do like to discover new music videos to tweet out and play on my Codebass Radio show.
\nSo, I started looking into it and found this on the Youtube-DL Github repo:
\n\nyoutube requires an additional signature since September 2012 which is not supported by old versions of youtube-dl. You can update youtube-dl with sudo youtube-dl --update
Grumble, grumble - that explains it....YouTube changed their requests in some way, and now it's ALL BROKEN! I'd be angry, but Youtube-DL enables something that YouTube doesn't really want, so you really have to expect this going in if you choose to take on a project that downloads videos like this.
\nI'm extremely impressed with YouTube-DL for fixing this so quickly though! So OK - I thought I'd just do an update and things would be fine....NOPE. See, while YouTube-DL works awesome now, the Node.js wrapper I was using does not work. Something about it is broken, but checking that git repo again, led me to another project by the same author called "node-ytdl". This one isn't a wrapper around the YouTube-DL python service...it's alllllll Javascript baby. Pretty cool, but this one didn't work either. It created a 0 byte file on my system, but that's about it.
\nI can't really blame the author of these tools for the brokenness, after all I see the last activity on the repo was about a month ago, and all this YouTube breaking went down sometime in September. At best, I can only say again that the original YouTube-DL is freaking... way on top of their game.
\nSo what can I do? Well it's time to get my hands dirty in Node.js and step away from the fancy wrappers. It's time to spawn a damn service myself! I pulled code heavily from the YouTube-DL wrapper (though I never did figure out why it was broken).
\nWe'll need a couple of requires to work with our external executable...
\n var spawn = require('child_process').spawn\n var split = require('event-stream').split\nNow, "event-stream" doesn't come free with Node.js, so do a "npm install event-stream" before working with it.
\nAnd then to launch our Python based YouTube-DL service:
\n downloader = spawn("youtube-dl",[myyoutubeurl], { cwd: myoutputdirectory } );\nLet's now add a listener for when it errors, and for when it completes:
\n downloader.on('exit', function() { console.log("I'm done"); });\n downloader.stderr.on('data', function(err) { console.log("I have an error: " + err.toString()); });\nAnd then lets spice it up a little more and add some logging from the service itself. We need to create that event stream "split" object and pipe our output to it:
\n downloader.stdout.setEncoding('utf8');\n var line = new split();\n downloader.stdout.pipe(line);\n line.on('data', onLineOutput);\n function onLineOutput(data) {\n if (data.indexOf("[download] Destination: ") != -1) {\n // found destination file\n myfilename = data.substr("[download] Destination: ".length, data.length);\n }\n console.log(data.toString();\n }\nSo in the above method, I'm just logging the line output to the console. It'll tell me if the file is found, give me some percentage complete info, how big the file is, how long it will take, the file name etc. I did a little string manipulation to find the filename as you can see about. It will say "[download] Destination: ", and what follows is the name of the file it's creating.
\nAnyway, that's the new downloader process I created! It's a little less sophisticated than my last one where I can choose the format I want (low quality, high quality, mp4, flv, or webm), but given that I got burned when YouTube changed their request model - I figure, keep it simple because it will change again...and I'll have to deal with the thing.
\n", "date_published": "2012-10-03T00:00:00Z" } , { "id": "https://benfarrell.com/blog/2012-09-28-this-boring-web-why-adobe-edge-isnt-for-me/", "url": "https://benfarrell.com/blog/2012-09-28-this-boring-web-why-adobe-edge-isnt-for-me/", "title": "This Boring Web - Why Adobe Edge isn't for me", "content_html": "And, of course by "me" - I do mean me....personally, and not really anybody else. This is evident from recent articles like "Does Adobe Finally Understand Developers" and "Has Adobe Gotten It's Mojo Back?".
\nThe Adobe Edge set of tools includes a lot of goodies that seem very solid for web developer type folks. I have to admit that Edge Inspect that enables live debugging on devices from a desktop might come in pretty handy one of these days. To tell you why this new set of tools isn't necessarily for me goes back to why I was originally an Adobe/Macromedia fan in the first place....
\nSee, whenever a new version of whatever platform/software came out, be it Director, Flash, even Photoshop, Illustrator or the rest - it was like an early Christmas. There was always something inside that new package that would let you do something that you could never do before. I remember when Director and Flash each got the ability to set pixels on a canvas. This was a huge deal - you could do some crazy things with this new stuff. And you couldn't do it on the web before that - so it was a world's first.
\nProductivity and good tooling was always a distant second to me. I'll be productive in whatever tools you give me, really. It was the magic trick I was always after - the "How'd you do that" reaction and the sense of wonder going in to a new and impossible problem.
\nI feel like that part of the web has kind of died in recent years, and I don't think it's because of Flash really, but it's because of the people touting good UI/UX, minimalistic design, etc. And, of course, its certainly not a bad thing to have a useable site - but I do feel like the tone of the web has changed, and you're not REALLY a web developer unless you are focused on clean UI.
\nTo me, a clean UI and useable site is quite important and something to strive for in your professional endeavors - but it's not where my passion lies. With browser fragmentation, the magic tricks we have up our sleeves don't work everywhere - and so we focus less on them and more on graceful degradation, and settling on boring everyday features in our web pages.
\nAnd so, the current state of things has led to "this boring web". No frills, no magic tricks, just clean, useable sites. Again, this isn't a bad thing - we NEED this type of thing, but it just doesn't fill my creative passions.
\nNow, Adobe Edge...
\nI think Adobe is listening to this new "boring web". This is what people want to do, it's what they will pay for. People want good tools, and Adobe has a history of building good tools. I think Edge will be great for Adobe's revenue, and will inspire a good many people to go out an create. But really....for me? A new open source font? A code editor? The ability to set break points in your responsive design to create media queries? This is all stuff we've been able to do for years without the fancy tools. While it may lead to more productivity - it doesn't inspire me personally to go out and create things that have never been created before.
\nAnother interesting thing to think about is that especially with tools like Edge Animate - Adobe is pushing animation and motion on otherwise boring old websites. Will there be blowback from this? A new generation of people will feel empowered to add new things they never would have thought to do before without these tools - but will the clean UI/UX and useability people revolt? I've read more than once (in so many words) that Adobe is turning HTML5 experiences into Flash. For goodness sakes, there are people complaining that no website ever should have video. You can't please everyone, of course, but I do wonder how these new tools will rub people the wrong way.
\nBottom line? I think Adobe Edge looks great and will be pushing some great new creative things forward. Hell, I'll probably give the tools a shot - I might like them. But it's nothing that you couldn't do before, so the Adobe "magic" is kinda gone for me. I guess instead, I'm looking for my magic from other places - like Chromium, Mobile Apps, and maybe even some C++ (god forbid).
\n", "date_published": "2012-09-28T00:00:00Z" } , { "id": "https://benfarrell.com/blog/2012-09-28-sweatin-to-the-web-nui-presentation-at-ncdevcon/", "url": "https://benfarrell.com/blog/2012-09-28-sweatin-to-the-web-nui-presentation-at-ncdevcon/", "title": "Sweatin to the Web - NUI Presentation at NCDevCon", "content_html": "Hello! If you're here, you might have come to my NUI (natural user interface) presentation at NCDevCon. NUI can mean anything from gesture input from your body, to speech control and anything that's natural and intuitive.
\nWe went NUI vs GUI in my presentation and the hardware behind it. We discovered how to develop with it using a few different Javascript solutions. In this post I'd like to link to a few things from my presentation for easy reference.
\nEarly not really NUI hardware
\n\n\n\nGetting NUI-ier
\n\n\n\n\nI also made mention of the Gamepad API for HTML5.
\nPrimarily we explored the Kinect SDK and OpenNI as the hardware support for the Microsoft Kinect and Asus Xtion Pro. We explored a few Javascript solutions for NUI:
\nI also went into a few rules from the Kinect for Windows Human Interface Guidelines. Finally, I demoed a KinectJS Kickstart I put together on Github, as well as my Kinect enabled music discovery project called Blastanova. Here's my presentation slides as well! Have fun with this stuff, it's awesome!
\n", "date_published": "2012-09-28T00:00:00Z" } , { "id": "https://benfarrell.com/blog/2012-09-11-wtf-html5-video/", "url": "https://benfarrell.com/blog/2012-09-11-wtf-html5-video/", "title": "WTF HTML5 Video?", "content_html": "Just thought I'd put up a quick little post this morning as we talked a little bit about one of our HTML5 video projects in our morning meeting.
\nHTML5 video is pretty rad, it's not all that though. Microsoft's Smooth Streaming + Silverlight and Adobe Flash's Media Server + AS3 give you the ultimate control (despite maybe being a little too complex for beginners). The AVPlayer framework in iOS is pretty nice too, but out of the box it doesn't give you nearly the same control as MS or Adobe.
\nAndroid is pretty weird. Its similar to iOS, but the HUGE catch is that you can only play one video at a time with it's native media player. Lets say you start up a stream, then pause it. You then try to start up another stream while that first one is playing. All is well. Go back to the first and unpause.....BOMB. It's broken and won't really explain why. Boo. It's because you can't have two media players running on Android at the same time. Something that any other platform lets you do.
\nHTML5 video looks fairly straightforward as you first get into it. On mobile it gets pretty funky - you'd pretty much assume it gets WTF. One such example?
\n\nSo you're humming along doing your awesome HTML5 video experience. Out of laziness or whatever, you don't CSSify your own player controls. You justify saying "I want that native experience on any platform - if it's on Android it needs to look like Android dammit, and god forbid I tweak Steve Job's master vision and override the Quicktime look and feel". So you're using the native player controls - quite easy to do by adding the controls="controls" attribute to your video tag.
\nThe day your project is due, some idiot marketer comes along and says "Hey, lets launch a little survey after they complete the video! We'll put a popup box over the video, and they have to click something to continue to the next". Your response, "Yah, ummm, whatever...sounds craptastic, but I'm not paying the bills". So you make a nice little CSS popup box, give it some absolute positioning and pop it right over the video.
\nYour marketing friend is all happy until they get out the corporate iPad. They finish the video, watch the popup, try to click a button.....and nothing. Your buttons don't work. "You must be mistaken you say...for this works in Chrome".
\nThey aren't mistaken - HTML5 video on iOS WILL STEAL YOUR DAMN TOUCH EVENTS. Anything intersecting with the video, whether it's under or over will not be tappable.
\nLet's rewind back to the beginning of our story - if you hadn't used the native controls and made your own CSS controls in the first place, this problem wouldn't exist. WTF HTML5 video?
\n\nThere is of course some more wonkiness on mobile, but surely not on our last bastion of browser freedom Chrome? Yes even Chrome.
\n\nIts a little unfair to blame Chrome yet, since it's like the only freaking browser capable of playing video. "But but but Firefox and OGG" you say. To that I say, find me an OGG file and I'll try it out, until then we're talking about Chrome. And oh yeah, IE9 and Safari I guess.....whatevs.
\nUnlike it's Android bretheren Chrome on the desktop can play 2 videos at the same time. Rock! However this just leads to extreme confusion when it can't. And your confusion is extremely likely when you're a developer. See Chrome CAN play 2 videos at the same time, just not 2 of the SAME videos at the same time.
\nYou can imagine this can lead to extremely confusing times as a developer, when you leave one tab open with your video loaded, and then try to launch another tab thinking you closed the first. You get nothing....a black screen. After pulling your hair out for 10 minutes trying to wrestle file paths in your code, you find the open tab and close it. Wheeeeeeeeeeeeeee it works again. WTF HTML5 Video?
\n\nLots of us work in an office, usually this means being kind to your cubical neighbors. So, either you wear headphones or you just develop without sound until you actually need to test it. I do this, and occasionally this means long stretches of time where I don't have headphones plugged into my desktop. I'm occasionally burned by this.
\nGuess what, if your machine doesn't have audio output setup in some fashion - the damn video won't even play. Seriously. I just tried this again in Chrome 21. If you want to play video, at least leave your headphones plugged in with the volume down. I haven't had this problem on my laptop since the speakers are built in and always there. But on a desktop, it can be pretty likely that you won't have any hardware audio device at the time.
\nWTF HTML5 Video?
\n\nThere are lots more WTFs, but I will leave you to wrap your mind around these.
\n", "date_published": "2012-09-11T00:00:00Z" } , { "id": "https://benfarrell.com/blog/2012-08-26-motion-control-fun-with-kinesis-io-and-javascript/", "url": "https://benfarrell.com/blog/2012-08-26-motion-control-fun-with-kinesis-io-and-javascript/", "title": "Motion Control Fun with Kinesis.io and Javascript", "content_html": "The Kinect is quite the contraption, I don't mean that in a negative way - I just mean that it's so full of hardware that works in tandem, it might be kind of hard to get a handle on what it fully does.
\nAt a very basic level you have both a normal medium res (640x480) camera and an infrared depth camera. Put these two things together and you have RGBD pictures! That's red, green, blue, and depth.
\nOther notable harware in the Kinect are the motors that move the cameras, and an array of 4 microphones which can supposedly pinpoint where sounds come from as they work together.
\nAltogether its fantastic, and makes me giddy to play. You can imagine (or look up) all the possibilities with this data. There seem to be a plethora of open source drivers to work with the Kinect (check out OpenNI). Though, what doesn't come with these open source solutions is the rocket science skeletal tracking developed by Microsoft that amazes me.
\nThis skeleton tracking is the RGBD data taken to the next level - it's obtained through the composite picture and a heavy amount of image processing and recognition. The Kinect SDK bundles these API methods so you can get any joint position you want. The Kinect for Windows demo application gives a nice little skeleton tracking demo which looks like this:
\n![\"skeletons\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2012/08/skeletons.jpg\")
\nSkeletal Tracking
You can get each joint position (x,y,z coordinates) through the Kinect SDK - MS simply connects the dots in these demos.
\nNow, I've recently been programming with a bit of C# (the language of choice for working with the Kinect), but as great as C# and MS Visual Studio is, developers who aren't really invested in MS tech - especially Javascript developers can feel a little left out. The great news is that there are a few Javascript solutions to work with the Kinect! I'm still evaluating the ways you can go, but one of them that looks great is Kinesis.io .
\nKinesis immediately starts at a disadvantage for those that want to take advantage of skeleton tracking. Since they are creating their own low level service that runs behind the scenes in Windows, I feel like they are probably rewriting all the rocket science work that MS did in their own SDK. This means that joint position data isn't available in the beta build that's available for download right now. They do however promise this and more in an upcoming build soon.
\nNow - what good is Kinesis if they don't offer skeletal data? Well - it seems like the Kinesis goal is really to allow you use of natural gestures and a hand-tracked cursor which more easily fit into your web workflow. Rather than allowing you to track full body joints - they are focusing on motion based useability that isn't so far off from what we do now with the mouse or touchscreen on a webpage.
\nThis sounds a little underwhelming from a hardcore geeky perspective, but it's a great first step - not to mention a REALISTIC step.
\nWhat does this mean? Well first of all let's take a look at how they handle a Kinect based cursor which you control with your hand. By adding the Kinesis.io CSS file to your project and initializing the Kinesis Javascript, you get some basic functionality that comes super easy.
\n![\"imagebox-demo2\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2012/08/imagebox-demo21.jpg\")
\nMotion Cursor Interaction
In the above screen shot you can see the Kinesis cursor (a hand) and a circle around the hand. Basically since there is no mouse click or even a touchscreen tap - the Kinect way (established as a user interface guideline by MS on the Xbox) is to hold your hand in position while a countdown timer is enacted.
\nKinesis.io gives you these elements in it's stylesheet. All you need to do to enact the pointer is to embed the CSS and Javascript file and instantiate it by doing the following in Javascript:
\nkinesis = new Kinesis;
Interestingly enough, even this little bit of code presented a problem for me. I'll admit that I don't know what the new operator does in Javascript if you don't use the parenthesis as in "new Object()" - however it's easy enough to see that you should do things this way from the Kinesis docs. What threw me though, is that to use that "kinesis" object, you can't custom name the variable - you absolutely need to name it "kinesis". I had just assumed any old variable name and scope would work. This cost me a couple hours of swearing until I finally tried copying and pasting the example directly.
\nWhen this is done, you get everything you need for the cursor! Clicking on things is a matter of marking your DOM elements as "interactive". If the CSS class is marked as interactive, the PARENT element will fire off it's click or href handler. This is important to understand - so....again: the element marked interactive must be INSIDE the thing you want to click. I'm not exactly sure why this is - maybe it's better semantic HTML. It does force you into wrapping your interactive element with something like an "a" tag like so:
\n
Once you have the Kinect instantiated and mark the correct elements as "interactive", you get some nice motion controlled interaction with your hand. If you recall I said you don't get skeletal data - you really don't, despite the fact that you have a cursor attached to your hand. Kinesis.io normalizes this data to a cursor without exposing you to the underlying joint data. In fact, either hand will control the cursor - you can switch off while you are interacting with your webpage.
\nThough I did say Kinesis.io normalizes the joint/skeletal data to a cursor - that doesn't mean you can't get position updates from it. Rigging it up is easy!
\nkinesis = new Kinesis; Kinesis.cursor = movement;
function movement(position) { console.info(position.x); console.info(position.y); console.info(position.z); }
\nAll we're doing here is supplying a callback function to the Kinesis cursor. Very simple! In fact, this is the basis for a few UI controls that Kinesis provides in their demos on Github
\nAll in all, it's very easy to work with the cursor - I will say that I did have a little trouble with it on my page. See, my body content is set to be 100% height. It's not designed to scroll. The Kinesis cursor (the hand) lives in a div element with a real height and width. Moving the cursor down to the bottom of the page suddenly made the page bigger as the baseline of the cursor extended the height to 100+ %. This caused my page to suddenly have scrollbars. Of course making the body overflow hidden via CSS fixed this problem.
\nI won't eat up space here with copied content from the Kinesis.io documentation. Needless to say, rigging up swipe up, down, left, and right gestures are super easy. Check the docs if you don't believe me! You have full control over which hand launches the event, too!
\nWhat's really great about the Kinesis.io SDK is that it offers a nice simulation tool. While Kinesis sounds like OSX is on their roadmap, you can't work on the Mac right now with the real hardware. Fortunately, the simulator works on both Windows and Mac. On Windows the simulator is pretty great so you can sit down and develop without needing to repeatedly run back in front of your Kinect to test things out. With the simulator, you can control the cursor, and click one of many buttons to initiate a swipe gesture.
\nThere are a few more buttons for gestures like leaning - unfortunately those are dimmed out and not available yet. Also, the speech detection UI on the simulator leads you to believe there's extra cool speech detection built into the SDK. Unfortunately going through the help forums, you find that it's not available yet.
\n![\"sim\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2012/08/sim.jpg\")
\nThe Kinesis.io Simulator
My first experience with the Kinect and Javascript was with KinectJS. While KinectJS seemed freaking awesome, it was much more DIY than Kinesis.io seemed. Also, KinectJS runs through an AIR app, while Kinesis.io runs through an invisible background service. I'll cover KinectJS another time - but Kinesis.io just looked way more promising to me. KinectJS hasn't been updated since February, all the while Kinesis.io is actively tweeting, blogging and working on their product as a for profit business. So I was excited to see something that showed promise of being updated regularly.
\nAdditionally, Kinesis seems to be working on their own UI components and really trying to standardize motion based interaction. For the Kinect to really latch on in the web standards world we live in, it needs to be easy to integrate with existing content on many different platforms. I really feel like Kinesis may succeed here - bringing motion control as a standard/common UI vehicle.
\nI really didn't have much bad to say about the Kinesis SDK while using the simulator. I had a few frustrating moments with the SDK that I've mentioned here. The UI controls they have on Github didn't seem immediately useful to my project - but I was able to copy some key functionality out from the Javascript class they provided.
\nOne example was the UIVerticalScroll component. How it was written didn't really apply to how I wanted to architect my app - but I was able to pull out code easily enough to do what they did. Unfortunately, I'm not exactly sure how useful this component is. At first glance - vertically scrolling via the position of your hand sounds natural. However - moving your hand back to a rested position will cause scrolling that you didn't intend. Likewise with raising your hand to then do a swipe gesture. These things should probably be thought out a little more! Not sure who the onus is on here....the developer (me) or Kinesis!
\nOnce I started in with the actual hardware - it got a little disappointed (yet still promising). I had a really frustrating time trying to get it working at first. Using the C# demos that come with the Kinect SDK, I can see that my skeleton tracks extremely well in low light and from a pretty decent distance. When I was looking at the skeletal output in my dark basement from 12 feet away it worked WAY better than I expected. Close to perfect in fact.
\nWith Kinesis, on the other hand, my basement seemed a little too dark, and it only seemed to work in a sweet spot of around 3-4 feet away from the device. Not only that but after running Kinesis for a little while, the cursor seemed jerky and gestures were almost impossible to pick up. I'm assuming there might be memory leaks in the beta, though I'm not sure.
\nI'm definitely looking forward to the new Kinesis drop right around the corner. Hopefully we'll have joint tracking, speech detection, and performance improvements. Despite the hardware + the SDK being a little unusable after awhile now, I'm still jazzed about Kinesis and what they are trying to accomplish. I'm highly psyched about their latest build and hope it comes soon! In the meantime however, I'll take another peek at KinectJS because I just don't know how long I can hold out for full skeletal tracking!
\n", "date_published": "2012-08-26T00:00:00Z" } , { "id": "https://benfarrell.com/blog/2012-08-10-dont-tell-me-what-to-do-with-my-website/", "url": "https://benfarrell.com/blog/2012-08-10-dont-tell-me-what-to-do-with-my-website/", "title": "Don't Tell Me What to Do with my Website!", "content_html": "Don't Auto Play Video
\nLast week, a friend of mine re-tweeted something that made me cringe:
\n![\"tweet\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2012/08/tweet1.jpg\")
My friend's re-tweet had added a "This just in from 2001!".
\nSo look...the sentiment isn't a bad one. The snarky comment is full of snark, and I appreciate it. What I have issue with is "Period. End of story". That brief line right there encompasses all of my oogy feelings towards web development lately. I do realize this is just a tweet, and if I had a nice discussion with Mr. Russell we'd probably agree. But, nonetheless - here's what I'm hearing from Mr. Russell: "You should never do that on the web, and if you do - I know better than you".
\nI feel like I'm hearing this message from all corners lately. And I wouldn't tell Mr. Russell to stop saying it, instead I'd like to make a mental note to myself and web professionals in general to develop a thicker skin, listen and evaluate according to needs of your audience and clients, but don't succumb to peer pressure.
\nSo, "Period. End of story"? No, that's not the end of the story. In fact, this directive from Mr. Russell is completely inapplicable to my work. I spend most of my day dealing with online video. I don't work with Hulu, but Hulu is a good example. If I deep link to a show or episode or clip, you're damn right I want it to autostart. That's the whole point of Hulu. You're there to watch TV. Having to take an extra step to click play is not the best for useability in this situation.
\nOn the other hand, if Hulu were embedding their clip on someone's blog - then yes, it's wise to not auto-play the video. Especially on mobile, it's standard practice to conserve bandwidth - so auto starting a huge bandwidth suck isn't desirable.
\nThe message here is to evaluate the needs of your audience and weigh carefully what to do. It sounds like Mr. Russell has a different audience and needs than I do - which is perfectly fine. I'm also sure that some splashy mini-site drove him over the edge with a 1080p HD video that wasn't necessary.
\nWe can all vent - it's cool :) Just make sure to have a thick skin when you listen to other's rants.
\n\nDon't use Flash
\nThere is a follow up rule to the "Don't use Flash" rule, and that of course is "Don't do web site intros". They aren't mutually exclusive of course, but the "skip intro" sites heavily weigh towards Flash.
\nFirst lets talk intros. Yes, I agree - it's completely annoying to go to a restaurant's website and have an animation with smooth jazz. All I really want to do after all is look at their hours, find their number and make a reservation on my mobile device. The Flash intro gets in the way here - and it's super annoying. This is why you'll hear lots of "Death to skip intro sites!".
\nGuess what though, just because someone's ranting doesn't mean you should listen. Adrian Pomilio (@adrianpomilio) recently pointed out to me a great blog post called "The Difference Between Art and Design". It really has some great insight into what you achieve when you apply artistic motivation to something versus when you put your UI designer hat on.
\nThe trend these days is towards useable sites. It's a good trend. We need more useable sites in the world. Reading text is great for instant information, and having a UI designer/architect get me where I want to go faster is fantastic.
\nThat said - perhaps the point of my site isn't to convey information. Perhaps I'd rather have my site convey emotion and feeling. Probably one big mistake of the "skip intro" restaurant owner is that being someone who likes to impress with their atmosphere and good food is that they are in the business of selling emotion and feeling - and this carries over to their site.
\nIt's not wrong to want to do that, but they need to evaluate why people are there and what they want to do.
\nPerhaps an intro would work better on a car companies site? Car commercials regularly try to evoke power, speed, and sleekness. Often when shopping for a car, I don't find myself visiting Nissan.com, I find myself looking for local car dealers. Therefore, the local dealers sites should be useable - however, what about Nissan.com? Here I might really want to sell the brand and evoke feeling rather than provide useful information. I'd want to create a close tie with what I do in my TV commercials with what I aim to accomplish on my site. So here, an intro, a game, or immersive interactive experience might be perfectly appropriate.
\nOf course there may also be information to convey - so the emotiveness of the website must be balanced with useablility. Now I ask you, why is a "skip intro" button not appropriate here? Because someone on Twitter made fun of them? Don't listen - have a thick skin, and defend yourself with well reasoned, articulate arguments based on the audience you are trying to reach or the client you are trying to please.
\nIs Flash wrong for doing such a site? No - as long as you have facts to back up your decision. You need to know your audience and the tools. You need to realize that your Flash website won't reach an iOS device and perhaps not even and Android device. Do you care? Have you determined that the best place to reach your target audience is on the desktop?
\nFor these expressive sites, Flash is probably the best tool for the job. Between 3D, video, and all of it's advanced capabilites you want to work across the highest percentage of browsers. It's a good tool for this (right now). Does that mean that you won't get made fun of somewhere on Twitter because "flash sucks", or "flash is using all my CPUs", or "use modern open source technologies"? No - you're pretty much asking for it with this crowd by using Flash. And for every 1 angry open source nerd that blocks Flash, there are probably 50 less tech savvy that don't know or don't care. Ask yourself - who are you trying to appease?
\n\nDon't use Comic Sans
\nI think it all boils down to this. "Don't use X overused thing". This could be anything really. If it's overused, there's probably an angry nerd knocking it. And that's fine - angry nerds will knock things. Hell, I knock things too. I don't like Apple for example. But guess what, i have to put up with them because for the short term they are popular, and for the time being they can be the best way to reach an audience. I just have to know my audience and figure out what's best.
\nOne thing I've been guilty of knocking in the past, and went along with the angry pitchforked mob was the use of the Comic Sans font. But why? Its just a font, like any other. Sure it's overused - but why is that? Well, let me answer - because it's a decent font. It's easy to read and just a little playful. If I told you I had a fun quirky font for you, would you stab me with your pitchfork?
\nThe thing about Comic Sans, though, is that enough angry nerds seem to have gotten enough non-tech savvy people on board. So now the font seems widely despised. Does that mean you should never use Comic Sans? Hell no - but you have to be careful. Sure if you design a poster or Word Doc for grandma with comic sans she'll think it's a delightful treat - and really who are you designing this for? It's for her, so as long as she's happy it's cool right? Maybe, but chances are she'll have some angry nerd nephew that will convince her it's shit. Comic Sans hate is so pervasive in our web culture, there seems to be a meta-audience of angry nerds that you may or may not have to worry about.
\nThey aren't your TARGET audience, but if they catch wind of what you did - you'll get plastered all over twitter. And not in a good way.
\n\nShould You Worry About the Meta-Audience?
\nComic Sans is a pretty extreme example, and it's easy enough to avoid. So I dare say it might be good to avoid it. I'd never say never though. More often than not, those who use it probably just don't know about the undercurrent of hatred for our fun friend.
\nLess extreme examples include all of the above in the post, and of course, a hell of a lot more. But some are pretty hard to avoid using/doing (and yes I'm assuming you have a good reason for using/doing them). The best advice is to know and research your audience well enough to make informed decisions. That advice is ingrained in all of us who do this kind of work.
\nI will pose to you, though - you also need to know your meta-audience as well. Will those angry nerds get angry over what you did? Will they sabotage your project by persuading your target audience? How do your audience and meta-audience interact?
\nStuff to think about. Best way to shut down angry nerds though? Thick skin, well reasoned arguments, lots of research, politeness. These tools, if used effectively should minimize the impact of your meta-audience.
\n", "date_published": "2012-08-10T00:00:00Z" } , { "id": "https://benfarrell.com/blog/2012-07-23-injecting-metadata-with-ffmpeg-and-node-js/", "url": "https://benfarrell.com/blog/2012-07-23-injecting-metadata-with-ffmpeg-and-node-js/", "title": "Injecting Metadata with FFMpeg and Node.js", "content_html": "In a previous post, I told you guys how awesome FFMpeg was. I shared how to use it to convert video files using the Node.js FFMpeg wrapper.
\nGiven all of the awesome things that FFMpeg does, injecting metadata into a file doesn't sound that impressive - but it's an important task...and one that I struggled with a bit! Why the struggle? Well, its the syntax!
\nUsing FFMpeg on the command line for this actually is a piece of cake:
\nffmpeg -i in.mp3 -metadata title="my title" out.mp3
What's happening here isn't hard to see - we're taking an mp3 called "in.mp3", setting the title field in our metadata, and writing our "out.mp3" file. Really easy, right?
\nWell, let's put this in Node.js terms. Previously, to transcode/convert my mp4 video to mp3 audio, I simply executed the following in Node.js:
\nffmpeg.exec(["-i", infile, outfile], callback);
Simple right? Well it's pretty easy to add a flag to make it do something extra:
\nffmpeg.exec(["-i", infile, "-myflagtodosomething", outfile], callback);
Indeed it is easy - flags go between the infile and outfile array elements. You can have as many flags as you want. So lets revisit our metadata flag. We see that the flag is called "-metadata". But there's more to this flag of course - there's a secondary option called "title" AND I have to use this as a key, and set the value of this key to be the actual title I want.
\nHow to do this? Well documentation calls everything you put in that array a "flag". So I'm thinking, either non flags belong outside, or everything goes in one flag as a big string or something....like "-metadata title="my title". Turns out no. The way to do it is to treat the options like a separate flag, like so:
\nffmpeg.exec(["-i", infile, "-metadata", "title=mytitle", outfile], callback);
So this works great! Though, I have another problem - it seems like it's processing a fair bit and running the whole file. My formerly 128kbps MP3 file has been reduced to 64kbps in my output! Lots of lost quality here unfortunately. However, we can tell it to not process the audio and leave as is - just simply copy the audio data to the new file:
\nffmpeg.exec(["-i", infile, "-acodec", "copy", "-metadata", "title=mytitle", outfile], callback);
Perfect! Audio is intact. Our "acodec" flag specifically targets our audio track. I'm not worried about video here since it's an MP3 audio file.
\nLast thing...since I'm simply adding metadata to an existing file, can't we just use the same file? Can't our in.mp3 be the same as our out.mp3? Yes, but it's tricky. If you simply execute the command line FFMpeg option, you can use the same file for in and out. However, partway through this process, FFMpeg will ask if you'd like to overwrite the file. You hit "Y", and it continues. Try to do the same thing in Node.js - well it hangs. The process locks up waiting for you to hit "Y", with no way to continue. Bad news....
\nGood news is that you can use a "-y" flag!
\nffmpeg.exec(["-i", "myfile.mp3", "-y", "-acodec", "copy", "-metadata", "title=mytitle", "myfile.mp3"], callback);
The "-y" flag forces the process into choosing yes to overwrite the file.
\nSo there it is - adding metadata with Node.js/FFMpeg. And since it's simply a process that's spawned, I imagine this applies equally well to any other language - be it Python, PHP, Ruby, whatever.
\nGood luck with your metadata!
\n", "date_published": "2012-07-23T00:00:00Z" } , { "id": "https://benfarrell.com/blog/2012-07-19-feedback-over-time-a-hard-skill-to-learn-for-software-devs/", "url": "https://benfarrell.com/blog/2012-07-19-feedback-over-time-a-hard-skill-to-learn-for-software-devs/", "title": "Feedback over Time: A Hard Skill to Learn for Software Devs", "content_html": "If you possess this skill - kudos. Its one I don't, but I'm trying to learn. I don't mean to pigeon hole all software developers, but it's something that I imagine we have in common.
\nThe skill I'm talking about is working with SLOOOOOOOOW feedback. Results to your efforts that take days, weeks, maybe months. As software developers, we work with machines that give us results to our tinkering in a matter of milliseconds. You can tweak a system and instantly know what's wrong and what's right. You can correct the system in realtime until something works.
\nThe bottleneck here is you as a developer - you take time to work out how to fix/code the system and correct it so the output is valid.
\nWhat if the bottleneck to working with your results wasn't you, but the feedback process itself. What if the feedback process is so damn slow, you're left twiddling your thumbs waiting for results?
\nI wasn't aware that I'm not flexible enough to work like this until I saw a dietician. Now that I've realized I have this problem, it's pretty interesting working to try and correct it, and realize how it applies to other areas in my life.
\nMy wife would attest to my willpower when it comes to eating snacks. I'm pretty good when I set my mind to a diet. But, I've been on so many diets over the years, and I've been at 230+ lbs for the longest time.
\nMy diet cycle went a little like this...
\nFirst I'd try a new diet that people are talking about, whether its veggie smoothies, no carbs, low calorie, whatever. More often than not, I'd lose 10lbs in the first 2 weeks with amazing feats of willpower saying no to all but the healthiest food. But unfortunately, it always started coming back after those two weeks. When the weight was mostly back, I'd deem the diet a failure, ponder what to do for a week or two, and then start the next crazy diet.
\nIt was a process that always failed, and stupidly I thought it was the type of diet I was on. Maybe if I just found the one that worked for me, I'd be solid.
\nAfter many failed attempts, I just said "Screw it, let me just pay for counselling". I learned (or was rather reminded) right away that I'm really not losing 10lbs right away. It's most likely water retention from my salt intake. As I started eating healthier, my sodium intake would naturally go down resulting in massive weight lost. It fluctuates though, so of course, a little salt would bring the weight back, making me think the diet wasn't working.
\nI knew about water retention of course, but I was really stubborn enough to not see the immediate connection. After all, I see instant results, and I've grown to expect instant results from years of software development.
\nIt's only been a couple months - and we're shooting for .75lbs per week, so I've only lost 6 or 7lbs so far - however the change to my mindset is fantastic. Instead of focusing on immediate results from looking at the scale, I'm working on establishing better habits. The habit/behavior change is the thing I'm working on. It's a matter of shifting your results/feedback loop from the thing you are actually trying to accomplish (ie losing weight) to something that is easily observable. To get something easily observable - we create goals. Am I exercising at least 4x per week? Am I eating a planned breakfast at least 4x per week? Supposedly these goals, after meeting them for 6 months become habits in your brain.
\nAnyway - these days, I do watch the scale. But it takes a good couple of weeks to know what's really happening, am I trending up or down? However, if I can EASILY tell if I'm meeting my goals, and work based on that.
\nAnd now I have a skill I didn't have before - its still something that's hard to get used to, but it does broaden my horizon. In my opinion, its a skill that easily translates into running a business or marketing, and is a pretty core thing to learn if you'd like go beyond software development.
\nWe're bombarded with overnight successes in the tech industry - only behind the scenes, it's really not overnight. Instead - these entrepreneurs spend countless hours toiling away and building their brand. Marketers spend lots of time managing a campaign. Guess what? Feedback isn't easily attainable here. Instead - they probably create some goals that they perceive will help them in the long run. Day to day, they stick to these goals - and if results aren't achieved way down the line (after months), they tweak. The secret here, is knowing your market well enough to create good day to day goals. Luckily for my weight loss, I have a dietician that can help me out.
\nAmazing what a little patience will get you I bet! Its a skill I look forward to practicing in my future endeavors.
\n", "date_published": "2012-07-19T00:00:00Z" } , { "id": "https://benfarrell.com/blog/2012-06-28-showing-an-image-for-your-audio-only-hls-stream-on-ios/", "url": "https://benfarrell.com/blog/2012-06-28-showing-an-image-for-your-audio-only-hls-stream-on-ios/", "title": "Showing an Image for your Audio Only HLS Stream on iOS", "content_html": "Now that I have a handle on this, it seemed like a sick joke how many false paths I was led down with this task.
\nSo what am I trying to do here, what's the use case?
\nWell - one of the unique things about the Apple marketplace is their restrictions on streaming network video. If I have an Apple HTTP Live Stream that I host on my site and I'd like my iOS device to stream/play it Apple has some restrictions. If I don't follow these restrictions, they will reject my app!
\nOne of these restrictions is to have an audio only component to my stream that is 64kbps or less. In the world of streaming, you'd typically switch back and forth between different quality streams depending on your bandwidth. And in this case, Apple forces you to have a low quality, no video stream. The reasoning is that if the user has a bad connection, they'll sit through this instead of a video that keeps pausing to buffer.
\nThis presents an interesting user experience problem. Does my app feel broken if my stream keeps stuttering and buffering, or does my app feel broken if I see no video, but a black screen and just hear audio. Personally, as someone who is tech savvy I know a bad connection when I see it, so I'd rather have it pause to buffer, but ANYWAY....
\nOne of the ways to enhance this unfortunate user experience is to show some sort of text to indicate to the user that "No this app isn't broken, we're just not showing you video because your connection sucks". How to do that though? Well it was suggested to me on the Apple forums to embed an image into the audio only stream, so that when it switches over the user is presented with the messaging.
\nHow to embed it in the stream isn't so hard and outlined here. Basically you set a couple flags to indicate that your metadata is a "picture" and where the file is that you'd like to embed. Just be careful though, because the image is embedded into every segment and contributes to your 64kbps allotment.
\nShowing the image is the thing that I had a hard time with. The first thing that led me astray is this line from the HTTP Live Streaming Overview:
\n\n\nIf an audio-only stream includes an image as metadata, the Apple client software automatically displays it. Currently, the only metadata that is automatically displayed by the Apple-supplied client software is a still image accompanying an audio-only stream.
\n
Well FANTASTIC right? So if I included an image in my stream as metadata, it will display automatically! Um no. And let me tell you, it was pure hell trying to find this out. I was trying to test my newly minted stream outside of my app. I used Safari, Quicktime, and VLC to try to see my awesome image - no luck. I even opened the AAC file in Adobe Audition - it wasn't there either in the metadata section.
\nThen I started my online hunt for example streams online with this image. No luck - either my Google-Fu isn't strong enough, or nobody does this (or advertises doing it). A co-worker pointed out that he DID see the image data in the stream, and yes, my AAC audio segments were the right size to have the image in them, so what gives?
\nTurns out that this magical image only works using the video tag in HTML on iOS Safari and in your own iOS app. So there's no hope in verifying it on the desktop. But wait, it gets worse!
\nIn your own iOS app, it will only display automatically if you are using the older MPMoviePlayer framework. I checked it out, and yup it worked (finally)! The problem is that I was using the newer AVPlayer framework. And from here on in, it's COMPLETELY manual. What's manual mean? Well, you're in charge of all your metadata. This means that if you'd like to show that image you embedded, you need to grab the raw binary data, convert it to an image, and display it your DAMN self.
\nFine, then - we have a "timedMetadata" property, let's use it:
\nfor (AVMetadataItem* metadata in [self._avplayer currentItem].timedMetadata) { if ([[metadata commonKey] isEqualToString:@"artwork"]) { self._overlayImage = [UIImage imageWithData:metadata.dataValue]; self._overlayImageView = [[UIImageView alloc] initWithFrame:CGRectMake(self.frame.origin.x, self.frame.origin.y, self._overlayImage.size.width, self._overlayImage.size.height)]; [self._overlayImageView setImage:self._overlayImage]; [self addSubview:self._overlayImageView]; } }
That's actually not too bad right? That timedMetadata property is pretty handy. There's one mind-boggling catch though. You must add a AVPlayerItem observer for timedMetadata, like so: (where item is a AVPlayerItem)
\n[item addObserver:self forKeyPath:@"timedMetadata" options:0 context:nil];
If you don't do this, your timedMetadata will be null. So it's like that old riddle - if a tree falls in the woods and nobody is around to observe it, did it really fall? Apple says no. Actually they didn't say no - they just assume that you'll arrive to that conclusion.
\nWhen you do add that observer, you'd think that you would have an event to trigger showing this image. That would be true.....if you don't care that it won't go away when your stream switches back to video+audio. It's kind of maddening.
\nSo, when you get the timedMetadata event, all seems well. You have the image data available to show the image, and you can go ahead and do it. After around 10 seconds pass and you get to your next segment you'll get another timedMetadata event. If the stream switched to video+audio, this will be the last one you get. It's kind of late to let us know that "for this past segment we should have not been showing the poster image".
\n"But don't worry", you might say - we'll just check the timedMetadata property of the AVPlayerItem. And I would say you're smart to try that, but no - metadata will always persist for this AVPlayerItem whether it's on the active segment or not. This means that with the timedMetadata property or timedMetadata events there seems to be absolutely no way to tell if the segment that you are currently playing has metadata on it and if it is an audio only segment.
\nIck. Well, what the hell is the point of the image metadata on an audio only stream if it's all manual and this hard to control. But I needed to persist to get this task done...how can we know when to show this image and when not to?
\nI tried with AVPlayerItem.tracks. This will expose a track listing for the asset. Seemed pretty good at first - I was noticing that it was showing me I had video, audio, and metadata. Occasionally video would be dropped, and this seemed to coincide with the audio only stream - however this wasn't always the case. It seemed very flaky - so in the end I couldn't base things off of the tracks listing.
\nFINALLY I found the AVPlayerItem.presentationSize. When the stream was audio only, it would indicate that the presentationSize.width and presentationSize.height were 0. And I can use this and ping my video every second to figure out if I should be showing my image to the user that the stream at this very moment is audio only.
\nWhat an experience. We've gone from the documentation indicating that the feature was automatic to having to wrangle our own bytes, manage everything ourselves, and deal with several weird quirks of iOS. I'm glad I'm drinking tonight.
\nThe worst part of it is, I got no help from Google searches and some limited* help from the Apple dev forums. So I hope this helps YOU!
\n*My limited help from the Apple dev forums consisted of a very nice developer hailing from Cupertino and another from Maryland saying he was having a hard time too. The Cupertino dev helped out immensely, but not enough because I don't think I was asking the right questions to get to my inevitable conclusion of suckiness.
\n", "date_published": "2012-06-28T00:00:00Z" } , { "id": "https://benfarrell.com/blog/2012-06-14-morph-your-media-with-node-js/", "url": "https://benfarrell.com/blog/2012-06-14-morph-your-media-with-node-js/", "title": "Morph your Media with Node.js", "content_html": "I've been loving me some Node.js lately!
\nOne of the most recent things I've done in Node was download and convert video while getting info about the video. All of this is possible in Node, though frankly, the credit ultimately goes to a mix of C++ and Python. There are some handy wrappers written in Node for these utilities.
\nLet's start where I needed to start:
\nYouTube-DL
\nSo, YouTube-DL is a utility written in Python and wrapped up in Node.JS. It allows you to give it a YouTube , Yahoo, Vimeo, or other video and either download it or get information about it. Why is this so hard? Well, sites have gotten pretty smart about protecting their content online. Though they typically use just a progressive/single video file served over HTTP, which is therefore publicly available, there are some protective measures in place. First, they obscure it. The YouTube ID you normally see gets translated to another unique ID for a completely different download site, probably on a content delivery network of some kind.
\nMore importantly, it looks like access to this content delivery network is time restricted. So say you actually use the YouTube site. You follow a link for a video, and that link is translated on the fly to a secret location in the Google vault. Also, the time at which you attempted to grab the video is encoded into some sort of hash. It might be good for an hour, a day--I'm not sure. The important thing is that I can't send you a link to the raw video file that you could open up on your machine later that week. So, the sharing of links is USELESS!
\nThis is the type of stuff that YouTube-DL deals with for you. They may be going through some sort of YouTube API or maybe not. I'm not sure. What I do know is that in Node.js I can write
\nyoutubedl.info(url, callback);
function callback(error, info) { }
\nMy callback will give me an error or info. If it's info, my info object will have a few things available by dot syntax like info.title, info.filename, info.thumbnail, and info.description.
\nThat's pretty cool. I can use it to build up metadata about what I'm downloading, but really, I ultimately want to download.
\ndl = youtubedl.download(url, outputDir, ["--format=18"]); dl.on('download', onDownloadBegin); dl.on('progress', onDownloadProgress); dl.on('error', onDownloadError); dl.on('end', onDownloadComplete);
And we're downloading! Note that the "format" flag isn't required. For my purposes, however, I actually do want to specify. See, I want to convert to MP3 (audio only). So I want the worst quality video but the highest quality audio. I have to be pretty specific in my demands. I did find the different format codes online. But I also put together a prioritized list of what I want:
\nHere, I prioritized the lowest video bitrates with the highest audio bitrates coming first. This should give me the worst video quality with the highest audio output. Pretty perfect for my needs. The problem, though, is that I can only request one format at a time. If the first one I request isn't available and I get an error, I just have to try again for each one in my list. If you don't specify the format, it's a little less painful, but you don't get to pick your priorities.
\nFFMpeg
\nFFMpeg is the bad-ass in your video toolbox. EVERYONE uses FFMpeg, and you should too. It converts,records, and streams audio and video of almost any type you can think of. The fact that I only want to scrape audio off a video file in my project is pretty much an insult to its awesome power. Yet, that's what I want to do.
\nWhen I originally tried to convert to MP3, it was a no-go. It turned out that I needed the MP3/LAME codecs installed. FFMpeg doesn't originally come with this stuff, because it is itself free and open source. MP3 isn't; it's proprietary. So rather than mix and match this stuff making FFMpeg a little tainted, we have to grab the libavcodec-extra-53 ourselves. I did it by doing an apt-get install on Linux:
\nsudo apt-get install ffmpeg libavcodec-extra-53
\nAfter that, I was free to use the FFMpeg/Node.js wrapper.
\nUsing FFMpeg is actually a pretty big topic, as there are tons of flags and settings you can use. In my case, it was really simple, though. I was a little confused on syntax within Node.js, but here's what I came up with that eventually worked:
\nffmpeg.exec(["-i", "myvideo.mp4", "myaudio.mp3"], callback); function callback(error, info) { }
Anyway, here we run FFMpeg, set the source as our video, and the destination as our mp3 file. Works pretty well!
\nMediaInfo
\nLastly, we have MediaInfo and its matching Node.js wrapper. MediaInfo gets information about the file you pass in. This includes TONS of metadata properties and file properties.
\nHere's how I used it:
\nmediainfo(fileref, callback); function callback(error, info) { }
There are WAAAYYYY too many properties that come back on that info object. Here are some that I use, but your mileage will vary. Just dig into the object and figure it out! It's actually nested in an array as well, so I'm including indices in my list:
\nSo there you go! We've gone from downloading a file through YouTube, converted the video to MP3, and then checked the file for duration and other metadata.
\n", "date_published": "2012-06-14T00:00:00Z" } , { "id": "https://benfarrell.com/blog/2012-05-26-node-js-with-jenkins/", "url": "https://benfarrell.com/blog/2012-05-26-node-js-with-jenkins/", "title": "Node.js with Jenkins", "content_html": "I've been messing around a lot with Node.js lately. The project I'm working on finally reached the point of being useful. As such, I need a stable build to work with and do my bidding.
\nI've gone ahead and made a release branch of my work in Git. And yes, it's easy enough to switch to my release branch on my dev machine and make a build - but all the same, I already have a copy of Jenkins up on my home server, so why not use it!?
\nBut using Node was uncharted territory for me in Jenkins - plus I'm not making a "build", I'm just running my Node.js script to read some RSS feeds and download content (OK it's more complicated than that, but that's the gist).
\nLuckily, there is a Node.js plugin for Jenkins. It's easy to find and install from the Jenkins management page.
\nOnce it's in there, how can we use it?
\nEASILY! Under "Add build step", you can choose "Execute Node.js script". Once you select this, you have a text area to type whatever Javascript you'd like to execute. Go ahead! Type console.log("hello world"); and give it a whirl. The build will execute, finish, and output "hello world" in the Jenkins log.
\nWaaaaaaaaait a second. I can do whatever I want? What about file system stuff? Yep....all your fs.whatever commands work. I was taken back a little by this - I had been sort of planning to use Apache ANT to kickoff a Node.js task all along, but what's the point? I can bypass ANT altogether and directly have Node do my work.
\nThinking about the implications of this, maybe I don't use ANT anymore for anything? Maybe I write a Node.js script to do the leg work of my build even if it's not a Node.js project. I like ANT, but markup can get a little unweildly sometimes, and I might rather script.
\nAnyway, back to what I was doing - my Node.js project. Like I said, my project reads some RSS feeds and downloads files. Now, how do I kick this off? Well it's as easy as requiring my file. My main Node.js file starts all the tasks I need, so requiring this file starts the job.
\nSo, in my Node.js/Jenkins text area I can type: discovery = require(process.cwd() + "/BuildAssetLibrary.js");
Now, apparently there's a "process" object in Node.js. I didn't know that going in, but it sure did help getting the current working directory of the process. This does the trick for pointing to my Jenkins workspace and running my JS files.
\nAnother thing to note when running a Jenkins task is to make sure you kill the process when you've done what you need to do. Normally in my dev environment, my task finished on its own, but I'd been noticing recently that it left itself running and doing nothing. Maybe that's my fault, maybe it's just a fact of life running a Node.js task. In the end, I found it helpful to run:
\nprocess.exit(code=0);
...and end the process. It's pretty important to make the task end, otherwise Jenkins will always be running it and never let you know when what you really want to do is done!
\nNow, here's the other thing - I have some Node package dependencies in my project. Node Package Manager (NPM) is sweet for grabbing all of these dependencies. Once it's installed on your Jenkins machine, you can use it to grab your project dependencies based on your needs that you've outlined in your package.json file.
\nHere's mine: { "name": "SharkAttack", "description": "music feed aggregator", "author": "Ben Farrell ", "version": "0.1.0", "dependencies": { "feedparser": "0.9.x", "jsftp": "0.2.x", "request": "2.9.x", "sax": "0.3.x", "http-get": "0.3.x", "xml2js": "0.1.x", "youtube-dl": "1.2.x" }, "engine": "node >= 0.6.x" }
So, with this package file, all I have to do is run "npm install" and this package is read and the dependencies are fetched. I can do this by adding a build step in Jenkins right before my Node.js build step. My build step will be "Execute Shell". And the command I give it is simply "npm install".
\nSo, in order:
\nOne last thing - I build a static config.js in my project. It's been helpful to centralize some major setting in my project like output folders. Since my build machine is different than my dev machine, it's even MORE helpful to have this configuration file. I can require the config.js file, alter any settings, and then run my Node.js project with the new settings.
\nHere's my complete Node.js build step in Jenkins:
\nconfig = require(process.cwd() + "/config.js"); config.directories.output = "/media/blastanova"; config.directories.assets = "/media/blastanova/assets"; config.directories.shows = "/media/blastanova/shows"; config.directories.show = "/media/blastanova/shows/SA001"; discovery = require(process.cwd() + "/BuildAssetLibrary.js");
Awesome right!? Needless to say, I'm happy to have Node.js in my Jenkins arsenal and it may turn out to be a handy ANT replacement if I have a need that arises!
\n", "date_published": "2012-05-26T00:00:00Z" } , { "id": "https://benfarrell.com/blog/2012-05-14-javascript-continuous-integration-with-google-closure/", "url": "https://benfarrell.com/blog/2012-05-14-javascript-continuous-integration-with-google-closure/", "title": "Javascript Continuous Integration with Google Closure", "content_html": "In less than a week, I'll be speaking at cfObjective! Why am I psyched about this, when in fact I don't use Adobe's server side tech Coldfusion at all? Well - see, cfObjective features a whole track dedicated to Javascript...they dub it jsObjective. Looks like I'm up there with some fantastic speakers!
\nIf you're short on time, already saw my preso, or whatever... here's some demos I showed:
\nUsing Type Enforcement with Google Closure and JSDoc style comments
\n\nLastly, here's my presentation slides
\nWhat is this that I'm speaking about? Well I'm talking about a workflow I adopted when working on some fairly large or fairly complex applications. I'm used to workflows where I can create a project with many files - all organized to work towards a company library. This works great when you compile everything down to a binary - but not so well with Javascript when we aren't in the practice of doing so.
\nWith Javascript, you write it and run it. While it can be pretty easy to tweak and then refresh the page to view your work, it's not the best for my workflow because when I complete a project I might have 10, 30, 50, or even more Javascript files that are in use in my project.
\nWithout any sort of compilation process, we're talking about tons of separate file requests, a large bloated <head> tag full of <script> tags, and a fairly decent thrashing on our browser and server.
So, why not compile our JS?!
\nHave you ever heard of Spacebag? Basically it's a product that features a clear plastic air-tight bag. You put your clothes in there, suck out all the air, and your left with a flattened bag that takes up much less space. It's easy to carry, organize, store, etc!
\nTo me that's just like Google's Closure! It takes your Javascript... all the different files you have...puts them together, and compresses it all down to be one tiny, organized, packaged file. You can easily email it, send it to a friend, and use it on multiple projects or websites.
\nWhat exactly IS Google Closure? It's rather unfortunately named actually, but only because there's two other Clo(j/s)ures in our web development world. First there's the concept of a Closure. It's commonly described as a function in a function, but there's more to it than that. Worse, Closures are very core to the Javascript language, which makes working with Google Closure a little confusing.
\nThen of course there's Clojure - a language that runs on top of Java. Lots of people love the functional syntax of Clojure, and it seems to be the new hotness in the same vein as Python, Scala, and Ruby. Clojure even has a Javascript port which even further muddies up the Clo(s/j)ure waters.
\nGoogle Closure is a set of tools used internally on many of Google's project which have been open sourced to the wider world. In my presentation, I focus on one particular aspect - but to cover it all, here goes:
\nLet's look at some of the different ways to compile your JS via Google Closure:
\nTo see the compilation in action, go ahead and try out the Compiler Service web page. Even just using the example they have setup on the page is illuminating!
\nA side effect of the Linting is the warnings that the compiler will output for you. You can choose to ignore them if you like, but they can be pretty useful. I left a semi-colon out once that would have made IE fail, and the Linting caught it saving me from hours of frustration.
\nYou can go further though. Using JSDoc markup lets you type your variables, methods, and parameters. As you probably know, Javascript is not a typed language. This means that I can have a variable "var myVar = something" and put whatever I want in there...a string, a number, an object, whatever. Javascript won't complain. Compare this to a typed language where "var myVar:int = something" is used. Here, if you set "myVar" to a string and it's expecting an integer, your code will likely produce an error.
\nIt can feel a little freer to be able to put whatever we want in, and it is! However, a lot of the time you're using one variable as one type of thing. If you accidentally pass another type of thing to that variable you might get unexpected behavior even though it doesn't crash.
\nFor example x = y + z where y=1 and z=1 would make x = 2...right? Well what if y="1" and z="1" (strings)? This would concatenate the strings and make x="11" - Not what you were probably looking for! Javascript wouldn't alert you that you were doing something funny here, but you can use JSDoc to make some comments that y is an integer, and so is z:
\n/** @type {number} */\nvar y = 1;\n/** @type {number} */\nvar z = 1;\nThis type of notation wouldn't be picked up by Javascript, so wouldn't affect your code performance at all (it's just a comment after all). It WOULD get picked up by the Linter. If you passed one variable of type number to a method that only accepted strings, you'd get an error!
\nCheck out this GitHub repo for a demo
\nHere's a little bit about the different types of comments you can use:
\nhttps://developers.google.com/closure/compiler/docs/js-for-compiler#types
\nAnother side effect of using JSDoc is that you can also use the JSDoc Toolkit. This is a separate process which is outside of the Google Compiler, however since you've already done the work of marking your code with JSDoc, you can automatically use the toolkit. Basically, this is another "compilation" process. It doesn't change your code - but it does read it, and create some nice HTML based documentation for you that describes how your code works.
\nThe Google Compiler strives to follow the JSDoc Toolkit specs exactly, so you can be sure that what you comment will work in both the Google Compiler and the toolkit at all times.
\nGo ahead and check out my examples on Github - they all generate JSDocs!
\n\nThis theory is all well and good, but how to we actually run the compiler? The Closure Compiler is a Java JAR, so you can use the compiler commands against the JAR file, but I'm not one for memorizing commands and constantly typing.
\nInstead, I like to start with an ANT file. Apache ANT is a Java based build tool. It automates our process. We tell things how we'd like to build using an XML like syntax, and then run the ANT build file.
\nMy very first Closure Compiler project was a music synthesizer that builds some music theory off of the awesome Audiolib.js project.
\nAnyway, here's my ANT build file that both compiles my project and builds documentation around it using JSDoc Toolkit:
\nhttps://github.com/bengfarrell/synth.js/blob/master/build.xml
\nWhy stop there though!? We have a decent build script in place. To use it, just type "ant" into your command shell in the project directory. ANT assumes you mean to use "build.xml" as your build script and goes on it's merry way creating your build.
\nWhat if I don't want to use the command shell, but wanted to use a nice web interface to build my project? Well that's where Jenkins comes in. Jenkins is a "Continuous Integration" system that runs task from a web application shell. What's so "continuous" about it? Well that's the cool part! Jenkins, in addition to offering a simple one-click button to run your build, can integrate with SVN, GIT, and other source control management tools. We can tell it to run our Javascript project every time somebody checks in new code. If the Google Linter fails, it will produce an error, marking the build as broken. The person who broke the build, and anyone subscribed, will be notified immediately what actually happen. It's pretty sweet!
\n![\"Screenshot](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2012/05/Screenshot-at-2012-05-13-214740.png\")
\nOutput from a Successful Jenkins Build
One thing I started getting into after getting a handle on Google Closure was Dojo. Dojo is interesting because it's an awesome toolkit, but makes the compilation process a little bit complicated. Some of the more modern and grandiose frameworks like Dojo are based on Require.js or something like it. What Require aims to solve is code dependency on the fly.
\nSay I have some code which automatically launches a pop-up panel. My panel will have a bazillion dependencies - and I don't want to include them as separate script tags in my HTML header. I also probably don't want to compile it into a huge Javascript file that I load at the start of my project. What if I don't use this panel right away? Why not load the Javascript when I need it?
\nRequire.js does just that - when you require a dependency, it will check if it's loaded. If it's not loaded, it will load it then use it. Dojo is heavily based on Require.js so we can talk about both at the same time here.
\nSo you can leave things alone here, and not use a compiler at all. Tons of scripts will just load on demand, and I'm sure many projects are done this way. You can be smarter though! Dojo has it's own build process to handle smarter loading. You can group Javascript files and functionality together into what's called "layers". You would then load these layers on demand as you need them.
\nFor example, if you had an address book in your application and also a stock widget - you might group all of your address book Javascript files and widgets into an "addressbook" layer. Likewise with the stock widget. Since you don't use the stock widget at the same time as the address book (and you might never use the address book at all) it makes since not to load these modules until you need to use them.
\nRequire.js will function the same way - it will load up the dependencies. However, now, the dependencies just happened to be packaged up differently. Require.js and Dojo still knows where to look though!
\nBut wouldn't it be a nightmare to have to know which files have which dependencies? If those dependencies have other dependencies, and so on and so on? The Google Closure compilation process requires you to know which files to package - but with Require.js you just don't know the files you need unless you spend hours and hours researching every little bit of your code and framework you use.
\nThat's why Dojo has it's own builder. You tell the build profile which files you want in which layers, but the good thing is that you don't have to worry about dependencies - thank goodness. The Dojo builder will figure out all of that for you.
\nBut it all comes back full circle! The Dojo builder uses either the Mozilla Rhino compiler or the Google Closure compiler (whichever you choose).
\nIt gets a little complicated when you get into Require.js, but the Google Closure stuff is pretty easy when you keep it simple. Check out my examples on github to gander at my build scripts and setup!
\nhttps://github.com/bengfarrell
\n", "date_published": "2012-05-14T00:00:00Z" } , { "id": "https://benfarrell.com/blog/2012-04-24-the-coffeescript-dilemma/", "url": "https://benfarrell.com/blog/2012-04-24-the-coffeescript-dilemma/", "title": "The CoffeeScript Dilemma", "content_html": "So, CoffeeScript is pretty darn interesting. A couple years back when I made a concerted effort to take Javascript more seriously as Flash's stranglehold grip on the UI industry was nearing an end, I didn't like Javascript.
\nIt felt quirky and weird, and messed with my OO view of the world. Contrast that to Java, which I've been picking up a bit lately for Android development - and I feel right at home with that. I think a lot of people feel this way when it comes to Javascript. To me, it was an acquired taste.
\nI do dig JS now, and there's tons of little helpers you can use to make it fit whatever workflow you choose. I'm really digging on both Dojo and Google Closure now. There's tons of stuff out there.
\nBefore I liked Javascript, and it was still a bit frustrating, I strongly felt that JS would go away in 7-10 years - it would simply be a compiler target. Both Google Dart and Coffeescript were available, so it made sense. JS would be the "assembly language" of the web.
\nI don't know that I feel much different now about Javascript - except now that I use it, and feel it's not frustrating anymore, I do feel like the language will survive more than I gave it credit for. As in, maybe for lots of application, Javascript may be phased out in favor of things like Google Web Toolkit (Java), Dart, and possibly Coffeescript.
\nMeanwhile for other small projects and various snippets on your web page, JS will be in full effect.
\nThe Implementation Phase
\nWhat I didn't daydream on, was the implementation phase as these various languages see the light of day and gain more traction. I said hey, "We're using Javascript now - and in 10 years something else will be popular" (though still compiling to Javascript). I didn't even think about what happens in between.
\nI think we're starting to see this answer in the form of a dilemma in regards to CoffeeScript.
\nI Don't Really Like CoffeeScript
\nThere I said it! It's probably no surprise to those that know me that I don't really care for it. But I did go ahead and learn a little about it, and why someone would use it. The best pro-CoffeeScript argument was this 44 minute video from 37Signals Sam Stephenson called "Better JS Through CoffeeScript".
\nThe arguments presented in the video were largely "Look at how sexy this code is", "You save so much typing", etc etc. The underlying argument was that it's more readable.
\nWell, it's not....to me anyway - it was pretty hard to read, even as Sam was explaining it. I can definitely see if you have a Ruby background that it IS much more readable. From my perspective we're bringing JS into the Ruby comfort zone. I think, for me personally, Google's Dart would bring JS more into MY comfort zone.
\nAnyway, just keep that in mind. There is no "best" language in the world. There's just the best language for you and/or your job. And that's totally cool.
\nOne VERY compelling reason for CoffeeScript, on the other hand, was the whole compilation thing. Think about jQuery. If you do a jQuery "each" method, you're using the underlying framework to do something for you. It adds computational overhead.
\nContrast this with CoffeeScript - it will write better code out for you, and that's what your browser will be running. So that's a great thing in my mind - no framework overhead, and you get to write things that are easier for you to read and understand as a developer.
\nBut, again, to each his own. Maybe someday I'll get around to using Coffeescript and it will be another acquired taste.
\nThe Dilemma
\nI cracked open Twitter before lunch today and I see a comment thread on an Ember.js related blog post:
\nhttp://www.thesoftwaresimpleton.com/blog/2012/04/22/ember-js-routemanager/#comment-507972544
\nThe blog post itself was independently written (important for this conversation), and contained some how-tos for using the Ember.js framework,.
\nUnfortunately for me and @commadelimited (Andy Matthews), the examples were written in CoffeeScript, so they were pretty hard to read (for me anway).
\nThis might be where you expect me to rail on the author for using CoffeeScript....but no, that would be silly. This was written by a blogger in his spare time - we should be supporting him and appreciative regardless of if he wrote the example in Objective J.
\nOn the other hand, it really throws off my game! Part of me goes all American Redneck complaining why I have to press #1 on my phone for English.
\nUsing other languages both detract from and make Javascript awesome at the same time. Its so troubling that if I met this blogger in a bar and said "I will buy you a beer if you follow this advice"....I don't know what that advice would be. Obviously the dude likes CoffeeScript, why nay say?
\nDid we have these problems before? Usually, in my experience, we started at the language, and built frameworks on top of those languages). Now anything goes! And we need to mix and match.
\nI guess the best advice would be for me personally to learn CoffeeScript even though I don't plan to do things with it. It's a hard pill to swallow because learning has always equaled productivity for me. Now, it's learning for the sake of understanding our foreign language friends, who have something important to say.
\n", "date_published": "2012-04-24T00:00:00Z" } , { "id": "https://benfarrell.com/blog/2012-04-13-frontenders-vs-creative-devs/", "url": "https://benfarrell.com/blog/2012-04-13-frontenders-vs-creative-devs/", "title": "Front Enders vs Creative Developers", "content_html": "Today, Rebecca Murphey wrote a post called "A Baseline for Front-End Developers". The post had some good advice, it was well written, and it was well received, judging from the comments.
\nSomething about it really didn't sit right with me though. I was very uneasy about it, and I didn't know why until I thought about it on my bike ride home from work.
\nHere's the essence of the problem: THEY STOLE OUR WORD!!!!
\n\nFrankly, I thought I already knew. After all, the terms front end and back end have been bandied about for at least a couple decades now. To make sure I wasn't deluding myself, I turned to Wikipedia:
\n"In software architecture there may be many layers between the hardware and end user. Each can be spoken of as having a front end and a back end. The front is an abstraction, simplifying the underlying component by providing a user-friendly interface."
\n"In Web content management systems the terms front end and back end may refer to the end-user facing views of the CMS and the administrative views respectively."
\n"For major computer subsystems, a graphical file manager is a front end to the computer's file system, and a shell interfaces the operating system — the front end faces the user and the back end launches the programs of the operating system in response."
\nWhen I read the various definitions, one thing seemed very clear to me: a front end is any user-facing view. It is the layer of software that a human must go through to use a larger software or hardware system.
\nSo, by definition, a front-end developer must be someone who develops, scripts, programs, or codes a user interface. which is a pretty broad job description. By contrast, Rebecca gives a pretty narrow list of technologies and skills that a front-end developer must possess.
\nGuess what, though? There is nothing on her list but web technologies. More specifically, it's all about HTML, CSS, and Javascript. Maybe I'm right, maybe I'm wrong, but more to the point, we're talking about a very specific, though popular, subset of HTML usage. Rebecca isn't alone, she just seems to have encapsulated very well what I'm seeing in blogs, job postings, and more about the role of Front-End Developers. Honestly, I'd call what she defined as Front-End Developers (for lack of a better word), boutique web developers.
\nThe boutique web is a subset of smallish-scale sites that are quite attractively designed and use the latest HTML5 fantasticness. Its developers are striving for pixel perfect designs and may have some sexy animated jQuery widgets. Lots of money is being poured into this area right now given that people are running away from Flash . . . you know, the technology that we used to do boutique websites with.
\nI might be pigeonholing front enders by calling them boutiquers, but think about the following types of sites:
\nDo many of these skills overlap with Front-End Development? Sure. But we're talking definitions here. Developers are awesome and adaptable, so I can't pigeonhole them, but I can try to pigeonhole the term front-end developer. I think the baseline for front-end developers is geared towards boutique sites and only those that use HTML, CSS, or Javascript . . . and probably Ruby on Rails.
\nAgain, the people with these skills and passions are awesome, but it's a little confusing for people like me who have been developing old-school front ends since the late 90s. I really thought I could include myself in the front-end cool kids club, but I don't care about semantic markup or CSS preprocessors.
\nThere are TONS of other things I consider to be front end. There's Flash, Corona, Android, iOS, OSX, Windows, 3D games . . . the list goes on and on. Even inside the HTML/CSS/Javascript stack, there are other front ends I mentioned like games, video, and enterprise applications.
\nWhat do you call those developers? Well, lots of folks say, "I'm an Android developer", "I'm a C# developer", or "I'm a Corona developer." I find those to be a little narrow in scope, though. I definitely got burned calling myself a Flash developer when Flashageddon went down last November. I don't want to make that mistake again, tying my professional identity to a specific technology. Technology changes, evolves, and falls out of favor; unless you feel like rebranding yourself every few years, it might be wise to come up with a broader term for what you do.
\nBefore reading Rebecca's post, I thought front-end developer was a pretty decent broad term! I felt I could associate myself with it. After all, I really enjoy pushing graphics around and making users happy. But it turns out I can't, or rather, I don't really want to. If I'm reading Rebecca's post right, a front-end developer is more of the same. You're tying yourself to a specific technology, or rather, three of them. Yes, it's totally the hotness now, and it's been around for a while. You'll probably have a place in the world for the next 15 to 20 years if it's all you do and you keep up with the trends. But new hotnesses will inevitably come along, and HTML, CSS, and Javascript might get a little stale even if they're still relevant.
\nIf I don't identify as a front-end developer, what can I identify as?
\n\nSo creative developer is a term that I came up with on my own, only to find out it wasn't original in the least.
\nI occasionally listen to the Creative Coding Podcast. Both of the hosts, Seb Lee Delisle and Ian Lob, identify as creative coders. I have a feeling they dumb it down a little to be more approachable, but the perception I get from them is that creative coders sort of copy and paste code and tweak it by trial and error until it meets their needs. There are also folks calling themselves creative technologists. The impression I get from these folks is that they might be all sorts of brilliant but don't necessarily code at all.
\nI work with a bunch of people that call themselves software engineers. BORING. It evokes the right tone of capability, but I just don't prefer huge projects where I take data, manipulate it, and move it around. I like a little more razzle dazzle. I like getting in front of users, creating nifty things that people can see in front of their faces. I like doing things that make noncoders go "WOW!"
\nSo I took the creative part, since I like making creative applications, and I put something after it that evokes a little more confidence in building a software solution. Hence creative developer. But again, after I coined the term, I saw blogs and job postings for creative developers. Cool! But what IS a creative developer? What is our baseline, as Rebecca outlined for front-end developers?
\n\nI think that a baseline for our type of developer isn't so easy. You should probably know a lot, or at least not be afraid to learn a lot. Get to know a lot of different techs. Here are my suggestions:
\n\nFront-End Developers Are in Demand
\nWhat Rebecca said? Yeah, learn most of that. You should probably be a competent front-end developer because that is what's going on right now. Personally, I don't know in what universe I'd think CSS preprocessors are useful for my daily work. But whatever tech you don't learn on Rebecca's list, you should be able to talk intelligently about.
\n\nGet Linux
\nThe biggest realization I had last year was that if you use Windows, you're at a disadvantage. It's not something you can't overcome, but make things easier for yourself -- dual boot Linux. If you want to make Jesus cry, fine, get an Apple. The only thing making me wish I had OSX right now is that I need to switch OSs just to boot up Photoshop or Illustrator.
\n\nDo the Mobile Thing
\nYou should also do some mobile, of course. Phonegap is taking the easy way out. Useful yes, but learn some native development. Objective C and Java have their quirks, but they don't seem too bad.
\n\nDon't Forget Flash
\nFlash, on a technical level, is pretty awesome. AIR + Flash, like Phonegap, is taking the easy way out, but it's still awesome. I feel very fortunate that I already know Flash, because I don't want anything more to do with it voluntarily. Taking development advice from Adobe is like taking religious advice from Heaven's Gate. But there will be work in Flash for years to come. Who knows if the doomsayers are right and it's dying -- but it's still useful today.
\n\nThe Server Can Be Creative
\nI was so resistant to server-side tech for the longest time. After all, I'm a creative developer; I need some cool stuff on the screen to keep me occupied. I can do heavy lifting in the client -- it saves me from scaling my back-end architecture if people are doing something computationally expensive. Yes, clientside tech is slower, but it forces me to do some problem solving.
\nI was wrong and resistant for no reason. I dabbled a little with PHP just to get the job done, but it really wasn't fun. What's fun? Node.js is a freaking blast. People are so excited over it, and if you're not a server-side guru, everything in Node.js is new so there are no experts right now, just people having fun and getting awesome things done. It also has some very good conceptual parallels to other technologies like Python and Ruby. You know how the Node Package Manager works? Good, then you know conceptually how Ruby Gems and the Python Cheese Shop works.
\nSpeaking of Python, learn it! Or learn Ruby . . . but I'm seriously turned off by Ruby developers claiming they are god's gift. I hope those that do are the loud minority, but it's really off-putting. If I were a real creative developer, though, I would probably just get over it.
\nWhy is the server creative? Collaboration! Multiplayer games! Live video encoding! It's another tool in your arsenal. Also, the environment is a lot more mature than the clientside environment. People use languages in it that we really don't use on the clientside today. It will expand your horizons - - BELIEVE IT!
\n\nLearn C++
\nWhat is C++ good for? Everything. Remember what I said about learning the server because the client side is slower? You'll run into that less with C++. Chrome is leading the charge so you can run C++ in your browser. If you use Flash, you can use Alchemy (don't get me started on the Flash Premium feature debacle. Again, Adobe = Heaven's Gate). C++ will work on your Android phone, in your iOS application, everywhere. Bonus: You can learn leading creative coding frameworks like Cinder or Open Frameworks. A good basis in C++ will let you do some very cool, but very processor-intensive things.
\n\nUsers Aren't Everything
\nWith Flash and Javascript, I was stuck in such a rut. It felt like if I couldn't put it on the Web and let my legions of fans adore my work, it wasn't worth anything. Guess what? I don't have legions of adoring fans, and if I did, I should just freaking Vimeo or YouTube a shaky cam of my masterpiece.
\nThis mind-set prevented me from doing some cool things with some very cool technologies. Why use OpenFrameworks if I can't run it on the Web? Tacocopters are cool, but I can't give tacos to everyone who visits my site!
\nPlay around with stuff; you'll have fun and get some ideas. If you do anything cool, your fans will adore you with a YouTube video instead of a working demo in your browser.
\n\nThe Physical Side
\nSo there's the client side, the server side, and the PHYSICAL side! This is probably another term I'm coining that's in widespread use. Not everything you do has to be on a screen. If it does have to be on a screen, why not use a few projectors and a Microsoft Kinect? The Kinect is awesome, but explore Arduino as well.You can make some damn cool things if you don't constrain yourself to the screen and the Web. Remember, users aren't everything -- just put your masterpiece on YouTube.
\n\nTalk Big. Do Half of It.
\nThere are so many cool things to learn. A few of the above items are a to-do list for myself. I'm getting into native Android development, iOS development, Node.js, Python, and lots of Javascript. I've been meaning to do C++, Open Frameworks, Arduino, and creative installations for some time. I hope to have time and opportunity to explore these techs, but it might not happen. That's OK; there are just too many cool things to learn. Try to get to at least half the ones on your list.
\nShort answer: Nobody knows! By all means, be a front-end developer and reach Rebecca's baseline. You'll look good to HR people and be better able to answer the standard front end interview questions. You'll be good for several years, I'm sure. But for me, those skills ares only part of my passion. Frankly, I'd rather creatively develop.
\n(note: the link above is a replacement link to another link that died. Someone representing html.com contacted me to offer a guide on their site. Normally I leave 5 year old posts alone...but they were polite, insistent, and their link seems to offer decent advice)
\nThe fact is, we're seeing more smaller companies and startups bypass HR and recruiters. There are too many technologies to count -- if you don't know technology A, that's OK, you'll learn. The important thing is that you're passionate, driven, play well with others, and are proficient in something to prove you can be proficient in something else.
\nAlso, there's a developer shortage. You'll probably be juuuuuuust fine.
\n", "date_published": "2012-04-13T00:00:00Z" } , { "id": "https://benfarrell.com/blog/2012-03-06-the-long-and-short-of-a-simple-ui-component-in-dojo/", "url": "https://benfarrell.com/blog/2012-03-06-the-long-and-short-of-a-simple-ui-component-in-dojo/", "title": "The long and short of a simple UI Component in Dojo", "content_html": "Understanding Dojo is sorta like understanding Star Wars...you need the entire backstory to be able to figure out what's going on.
\nOn the surface, using Dojo elements are as simple as using HTML. You sprinkle a little bit of Dojo-ness in the div tag and it works. Pretty straightforward!
\nOn the same token, Star Wars seems like a pretty simple story of good guy fighting the bad guy to get the girl. It's only when you get the backstory, you understand why Luke and Leia kissing is kinda gross, and why Vader doesn't want to outright murder Luke.
\nWith Dojo, it's the same deal - if you don't understand the backstory, you're only going to get things working by sheer dumb luck.
\nSay I Google how to do a simple progress bar in Dojo. My first hit is this page. I even have code I can copy and paste that looks like HTML!
\n<div data-dojo-type="dijit.ProgressBar" style="width:300px"\n data-dojo-id="jsProgress" id="downloadProgress" data-dojo-props="maximum:10"></div>\nUnfortunately, if I put this on my page, it doesn't work. No I'm not dumb, I went to the Dojo download page and copied the script tag they provide in my HTML header:
\n\nBy all accounts, I've done what needs to be done, right?
\nNot exactly.
\nFirst of all, I might be thrown off by Googling "dojo progress bar" and coming up with dijit.ProgressBar. That's lesson the first!
\nDojo is divided up into 3 separate projects: Dojo, Dijit, and Dojox. Dojo is the center of it all. In it's (almost) 90kb core, it offers a lot of the same stuff something like jQuery would offer. The rest of the Dojo namespace has extra things which we can pull in as we need them (more on this in a bit).
\nDijits, or "Dojo Widgets", are the UI components to the Dojo team. ProgressBar, being a UI component, lives here in this namespace.
\nDojox features are the more experimental features. What's here might not be central to the Dojo vision, but pretty close - close enough to be shipped with the rest of Dojo.
\nWhat you should understand, is that all of these features put together weigh in at 5.3MB when you grab the zip. You might say that's too large to include in your project. You'd be right.
\nFortunately you don't have to force this on your users all at once (nor should you). The 90kb Dojo core is all you need at the start, but as you need the other features of Dojo, you can pull them in realtime.
\nIn other languages this might be an "include" or "import". In Dojo, it's a "require". So now the next piece of this puzzle, is to require the progress bar:
\n<script type="text/javascript">\n require(["dijit.ProgressBar"]);\n</script>\nFortunately for us, we can require the progress bar and anything the progress bar depends on is pulled in automatically because the Progress bar has its own include directives in the component.
\nSo now, we've written the HTML/Dojo markup to include the progress bar, and we've required the necessary classes to support it. But it still doesn't work!
\nThere's one last piece to this puzzle - the script tag. If you recall, I did say I wasn't an idiot - I did include Dojo.js here...something like this in my head tag:
\n<script src="//ajax.googleapis.com/ajax/libs/dojo/1.7.2/dojo/dojo.js"></script>\nIf we were simply using Dojo programmatically, this is all we'd need for basic functionality of a component...but we aren't! We're using some nifty declarative HTML markup, so there's one last step. We need to tell Dojo to scan the tags on the page and look for Dijit components. We do this by altering our script tag to be:
\n<script src="//ajax.googleapis.com/ajax/libs/dojo/1.7.2/dojo/dojo.js"\n data-dojo-config="parseOnLoad: true">\n</script>\nTelling our script to parse on load, means that when Dojo is loaded, it will scan and register all of our tags.
\nFantastic! Here's what we have so far:
\n<html>\n <head>\n <script src="//ajax.googleapis.com/ajax/libs/dojo/1.7.2/dojo/dojo.js"\n data-dojo-config="parseOnLoad: true">\n </script>\n\n <script type="text/javascript">\n require(["dijit/ProgressBar"]);\n </script>\n </head>\n\n <body>\n <div data-dojo-type="dijit.ProgressBar"\n style="width: 300px;"\n id="discoveryProgress"></div>\n </body>\n</html>\nUnfortunately, while our progress bar exists, and it technically works, it's looking kinda silly. That's because we haven't bothered to load our themes in yet. Without a theme picked, we get a barebones component which kind of looks like a broken wireframe.
\nAt this point, I'm going to go to my handy Dojo Theme Tester and pick the Dojo theme that suits me best for the immediate future. Claro looks good, so I'll use that. Not only do I need to include the Claro CSS and the Document CSS, but I also need to assign my body tag with a Claro class so that my entire document uses this theme.
\n<html>\n <head>\n <script src="//ajax.googleapis.com/ajax/libs/dojo/1.7.2/dojo/dojo.js"\n data-dojo-config="parseOnLoad: true">\n </script>\n\n <script type="text/javascript">\n require(["dijit/ProgressBar"]);\n </script>\n <link rel="stylesheet" type="text/css" href="http://ajax.googleapis.com/ajax/libs/dojo/1.7/dijit/themes/claro/document.css">\n <link rel="stylesheet" type="text/css" href="http://ajax.googleapis.com/ajax/libs/dojo/1.7/dijit/themes/claro/claro.css">\n </head>\n\n <body class="claro">\n <div data-dojo-type="dijit.ProgressBar"\n style="width: 300px;"\n id="discoveryProgress"></div>\n </body>\n</html>\nNow we're rocking! We have a Dijit Progress bar functioning on our webpage. Even though the point of this post is to simply get a UI component onto the page, I really can't leave it hanging at 0%. Let's go ahead and crank it to 80%.
\nFor this, we'll need a bit of script. For the sake of ease, I'm going to ignore all good sense and just put it in action in my HTML script tag.
\nWe're going to change our Javascript slightly - instead of a simple "require" directive, we're going to have it do something after the required thing has loaded.
\nThe simplest form of this is to write the following:
\n <script type="text/javascript">\n require(["dijit/ProgressBar"], function() {\n console.log("ready");\n });\n </script>\nThere are a couple of cool variations to this require script. First of all, you'll notice that we're requiring an array of Dojo components. Typically, we'll just want to listen for "dojo/ready" and perhaps a custom script and start our application from there.
\nSecondly, you'll notice that we didn't pass in any parameters through our function. Each index in our require list corresponds to each function parameter. So I could do the following:
\n require(["dijit/ProgressBar"], function(ProgressBar) {\nSee, when you require something, the parameter that comes through in the function is the constructor for that component. We can instantiate this object from this parameter!
\nHowever, what we REALLY want to do is tweak the progress bar we already have. So we need to find the existing progress bar on the page. You'll see two references in the Dojo docs to "byID". One is dijit.byId, the other is dojo.byId. The rule of thumb here is that if you want a reference to the DOM element, you would use dojo.byId, and if you want a component reference use dijit.byId.
\nUnfortunately, this is allllll old news in Dojo 1.7! It's very confusing - but this way of selection is in lots of basic documentation! Now, we use the "dom" object to find something in the DOM, and the "registry" object to find something in Dojo's component registry.
\nWe're going to need to step up our game here, and require the "dom", the "ProgressBar", and the "ready" component. In this way, it's kind of a lot to ask and checklist for basic setup, however once you get rolling, I PROMISE it gets easier.
\nOur new script tag reads:
\n <script type="text/javascript">\n require(["dojo/ready", "dijit/registry", "dijit/ProgressBar"], function(ready, registry){\n ready(function() {\n var myWidget = registry.byId("discoveryProgress");\n myWidget.update({ maximum: 100, progress: 80});\n });\n });\n </script>\nHere, we require all these necessary components. We then hinge the usage of our script on when dojo is "ready" and has loaded all that we required. We get a reference to the HTML declared component through the registry. From there we can update the progress bar as we want!
\nA full working example is here:
\n<html>\n\n <head>\n <script src="//ajax.googleapis.com/ajax/libs/dojo/1.7.2/dojo/dojo.js"\n data-dojo-config="parseOnLoad: true">\n </script>\n\n <script type="text/javascript">\n require(["dojo/ready", "dijit/registry", "dijit/ProgressBar"], function(ready, registry){\n ready(function() {\n var myWidget = registry.byId("discoveryProgress");\n myWidget.update({ maximum: 100, progress: 80});\n });\n });\n </script>\n <link rel="stylesheet" type="text/css" href="http://ajax.googleapis.com/ajax/libs/dojo/1.7/dijit/themes/claro/document.css">\n <link rel="stylesheet" type="text/css" href="http://ajax.googleapis.com/ajax/libs/dojo/1.7/dijit/themes/claro/claro.css">\n </head>\n\n <body class="claro">\n <div data-dojo-type="dijit.ProgressBar"\n style="width: 300px;"\n id="discoveryProgress"></div>\n </body>\n\n</html>\nSo there you go! A functional progress bar.
\n![\"progbar\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2012/03/progbar.png\")
It's unfortunate we had to go through such a long journey and learn so much to get to such a basic state - and that, I think, is Dojo's biggest weakness. If you choose to power through this initial learning and setup phase I really think you'll find Dojo to be absolutely fantastic.
\nThere will be the moments when you bang your head on the desk because you missed something really stupid or the Dojo docs weren't clear enough, however once you get into the rhythm of writing the code, it's a fun and productive toolkit to work with.
\nI hope this post and future ones will help with that!
\n", "date_published": "2012-03-06T00:00:00Z" } , { "id": "https://benfarrell.com/blog/2012-02-21-why-dojo-from-a-flashflexers-perspective/", "url": "https://benfarrell.com/blog/2012-02-21-why-dojo-from-a-flashflexers-perspective/", "title": "Why Dojo? From a Flash/Flexer's Perspective", "content_html": "Hey you guys... Is your Twitter stream bombarded with Ted Patrick tweets and retweets about Sencha? Do you get Sencha ads targeted at you when you visit websites, like EVERYWHERE you go?
\nI am. It's kind of annoying since I'm not looking to learn Sencha. Truthfully, though, you can hardly blame the current Sencha evangelist. After Adobe abandoned Flex sending it off to Apache, Ted seems to be leading a HUUUUUUGE marketing push trying to turn former Flex devs into Sencha/ExtJS devs. Ted is leading webinars, reaching out to people, posting on G+ threads, the whole deal.
\nI can hardly blame him because:
\na) I follow him on Twitter, so it's mostly my own damn fault
\nb) This is the perfect time to grab a bunch of disillusioned and lost Flex devs, and Ted is doing it in a perfectly ethical (although a bit overzealous way).
\nI looked into going "beyond" jQuery because of a G+ post by Jesse Warden claiming that ExtJS is the perfect compliment to enterprise development. He made the very apt analogy that Flash is to jQuery as Flex is to ExtJS/Sencha. It really made me re-think things because it had been drilled into my brain that jQuery "won", and given the tide of developers (84.8% of all websites using a library use jQuery) it seemed like there was no reason to use anything else.
\nTurns out, there really is a use for the lesser used libraries/frameworks/toolkits. We can see this because even if jQuery is and always will be your library of choice, as you build more advanced applications you might want some binding, templating, MVC, or other action. AngularJS by Google is getting more widely used. It gives you binding, declarative HTML markup for data, and other MVC goodies. There's Backbone.JS, Underscore.JS, and tons of others. A lot of folks seem to be combining all of these separate libraries together to custom fit their project. And I'm sure that works fantastically - I was doing similar things for a time.
\nAs these separate projects get more and more features, you may now have feature overlap between 3 different libraries you've used in your project. It seems a LITTLE wasteful at best. At worst, maybe your dev team does a very basic thing 3 different ways because binding is available 3 different ways given each of the libraries you're using.
\nWouldn't it be better, or even just worth looking into, if you could find a cohesive library/framework that offered all of these things together? Maybe it could even do a little more and help you make a massive Javascript application that loaded in components and scripts AS YOU NEED THEM instead of all at once in the beginning.
\nIt looked like Dojo and ExtJS were the most mature offerings in terms of what I describe. I took a look at ExtJS by Sencha first. At first glance, it looks pretty awesome. You've got designer and animator tools. They are a wee pricey, but considering my Flash background, spending $279 on Sencha Designer and $199 on Sencha Animator doesn't seem obscene.
\nWhat I can't get past is having to spend $329 for a Javascript library license! When I'm doing Flash/Flex development, I use the Flash Builder IDE for coding and sometimes Flash Professional if I'm doing animations or graphics. And that's cool. I chose to buy into these tools. I didn't have to pay to use Actionscript 3 though. My fellow developers who may not do Flash all the time can crack open any text editor and edit my code. They can check it back into SVN and contribute to the project. They can launch the Flex SDK command line and build my project. None of this costs a dime.
\nIf and when they get more invested in Flash/Flex, they might then want to buy Flash Builder or even IntelliJ for the advanced featureset including debugging, profiling, etc.
\nNot so with ExtJS. I'm buying a license to use the code base. Kind icky in my mind. If I start a project in ExtJS/Sencha and I pay my $329 for the seat, but then I want some more help on the project occasionally, each developer will then need to pay out $329 just to use/edit the code no matter what tools they use. If you are starting a major project, and plan the ExtJS cost into the budget, then great. But if you're a lone developer who wants to start something which blossoms into something bigger - the licensing can be a major roadblock.
\nMoreso, if I'm a consultant and I'm paid x dollars to develop a web application, I can easily choose and pay to use ExtJS. After two months, I make a deliverable. It might be the type of project where I do the major architecture and first phase, but then it gets shipped off internationally for maintenance mode or a lower cost second phase of the project. Well, if you use ExtJS, you've just upped maintenance mode by $329 for every developer that needs to touch the project. You've just cost your client lots of money.
\nThis is why I discounted Sencha and ExtJS and moved on to exploring Dojo. Dojo seems to offer MOST of what Sencha does. They both have a rich set of components, offer lazy loading of scripts so that your massive enterprise application doesn't load several hundred scripts (or a huge 2MB block of script) before the application starts up. Both have a build system - Dojo's runs on Rhino or Node, and is built on Closure or Shrinksafe. ExtJS seems to be built off of a Yahoo builder.
\nIn terms of scripting, both seem to nudge you in the direction of organizing your Javascript into classes which can be based on straight inheritance, or mixins.
\nBoth seem to offer a visual tool for design. Dojo's Maqetta is free and can be used for jQuery, YUI, or Dojo. I don't see myself being very productive in it though - I can lay things out, but it wasn't obvious to me if I could wire in data or do anything extensive. Though I haven't used Ext Designer, I have to imagine it's a better product than Maqetta. But then I seriously question how much I'd use a good tool for this anyway. It's like when everyone complains that design mode in Flex really sucks - but what if it didn't? Would you actually use it, or just code everything? I think we'd use it only like 5% of the time. But this is all personal preference - ExtJS wins here, but only if you feel like you need a good design tool, which I don't.
\nDojo seems to offer one thing that ExtJS doesn't though - and that's declarative HTML markup. This may be where my ExtJS ignorance comes in, so feel free to correct me if I'm wrong, but to mark up a component like you might do with Flex's MXML, you write a JSON object describing the component or layout you're introducing in ExtJS. This is pretty cool - but with Dojo, you get to write REAL HTML. I can tag existing divs or other elements as a Dojo component and they'll act like Dojo components. Yes, Dojo does offer templating, but you can choose which to use...or even to use both.
\nThe worst part of Dojo? They don't have a MASSIVE marketing and tutorial push. There is tons of ExtJS and Sencha material out there to learn from. Yes, there is tons of Dojo material, but much of it is older and not up to date. The material on the Dojo site is pretty technical and not very "quickstart-ish". What quickstart material they do have, don't seem to be big picture enough to inform you what's really going on. I always had bad luck copying and pasting code from tutorials and having it work without spending hours debugging. Most of the trouble with Dojo material is the switch to new loading routines that happened with the 1.7 release in December 2011. I feel like when they release 2.0 sometime this year, we'll have some better effort from the Dojo Foundation in terms of educating people on how to use the toolkit
\nAll in all, Dojo and Sencha seem eerily similar from everything I read. I feel so lucky that by learning Dojo, I'll probably feel right at home in ExtJS as well. For me it really comes down to the licensing that drives me to Dojo. If that wasn't an obstacle for me, there were more than a few times I've been frustrated by Dojo in the past couple weeks of learning it that I'd been open to abandoning ship if there was another choice. I get the feeling though that there are similar frustrations with understanding ExtJS (helped in large part by their tutorials).
\nThat's why I'm team Dojo right now! However, I'll continue using jQuery for the smaller projects. Dojo could easily replace jQuery, but for the smaller stuff when something minimal is all you need, there's no sense in going against the grain and using something that not many people have experience with.
\nI'll be posting more technical stuff on Dojo, but for now, this is how I got where I am!
\n", "date_published": "2012-02-21T00:00:00Z" } , { "id": "https://benfarrell.com/blog/2012-02-21-the-seven-stages-of-learning-dojo/", "url": "https://benfarrell.com/blog/2012-02-21-the-seven-stages-of-learning-dojo/", "title": "The Seven Stages of Learning Dojo", "content_html": "So, yah I'm learning Dojo, though from what I've heard this could easily apply to ExtJS. It's a different mindset to get accustomed to an enterprise Javascript framework, and comes with a bit of a steep learning curve.
\nAnd, so it begins....
\n"Ooooooh look at all these awesome features on http://dojotoolkit.org! They have a hot themetester, full documentation for like every single part of their toolkit. And GLORIOUS quickstarts!"
\n1. Shock and Denial
\n"WTF, I've copied this code EXACTLY, but it spews tons of errors, and doesn't even come close to working."
\n"I've looked up all these error messages, but I can't find a straightforward answer. Everyone references concepts I'm not familiar with....and they keep telling me to hop on an IRC channel to talk to the developers. Is IRC still a thing? Do they have a BBS as well? Maybe I can learn the language of my 14.4 kilo-baud modem and ask IT how to do stuff in Dojo by chriping and clicking at it"
\n"It's gotta be me....I'm an idiot, it can't be this difficult to get basic stuff working"
\n2. Pain and Guilt
\n"Screw, it - let's man up and just read massive quantities of documents and tutorials. Hell, I'm going to buy a book from 2008 even thought it's horribly out of date. I'm just going to sit through this initial pain to get a better understanding of the whole system."
\n"Am I really this dumb?"
\n3. Anger and Bargaining
\n"Dojo SUCKS, I hate it! It doesn't work. I need to be a freakin rocket scientist to understand this system"
\n"jQuery, will you take me back? Please? I just want some data stores, and some binding, and some well written and documented plugins."
\n4. Depression, Reflection, and Loneliness
\n"So lonely....the only materials I see online are from 2008, and from SitePen who charges for training, why would they help and give it all away for free?"
\n"Maybe if I had studied harder in school, this wouldn't be a problem"
\n5. The Upward Turn
\n"Going back to the beginning and reading a bunch of stuff seems to have paid off, I think I'm beginning to understand now"
\n"Where in the hell did they tell you about this parseOnLoad attribute in the script tag? It's not in any of the quickstart examples, but it's freaking mandatory to get anything to work"
\n"Well, no WONDER, this example didn't work, it was written in Dojo 1.6 before the AMD Loader, and I'm mixing and matching my code between 1.6 and 1.7 - and 1.7 has REALLY changed the game"
\n"Ahhhhhh.....I need to nest the requires, cause my require doesn't know what the ready object is before it loads the core stuff"
\n6. Acceptance and Hope
\n"Now that I understand the basics, I really like Dojo. It rocks pretty hard!"
\n"There are things I don't yet understand, but now that I have the big picture, I can roll with anything Dojo can throw at me, but all in all, I'm very psyched about all these new toys!"
\n7. Reconstruction and Working Through
\n"Dude, did I really just wire up a data store to a div which I made into a repeater, all inside a freaking accordion? Damn....awesome"
\n"I made a build system, and moved all my layers somewhere else, so I'm completely relying on my own custom build....I only get like 2 errors, and none of them block functionality. This will get better, but for now, it's awesome enough"
\nFin
\nAnd here I am. Though, I think I cycled through steps 3, 4, and 5 a few times.
\n", "date_published": "2012-02-21T00:00:00Z" } , { "id": "https://benfarrell.com/blog/2012-01-28-ios-video-is-cranky/", "url": "https://benfarrell.com/blog/2012-01-28-ios-video-is-cranky/", "title": "iOS Video is Cranky!", "content_html": "I'm close to wrapping up a decent sized iOS project, and my part was video playback. Prior to this, I've done tons of work with Flash video.
\nI hate to be labeled biased, but I've never appreciated the level of detail put into the Flash API before working with both HTML5 and iOS. Flash has tons of quality of service queries you can make, you have explicit control of bitrate on multibitrate streams, you have complete seek accuracy in streams. Virtually anything you could ever want, you got it!
\nCompare that to the AVPlayerFramework API for iOS. You can listen to status messages and put an observer on the current playhead time. That's pretty much it. And those status messages come in the form of ready to "play", "failed", "played to the end", "failed to play to the end" and "unknown".
\nThe lack of features actually isn't too much of a problem for streaming playback. Streaming content is handled like a champ by the framework. A bad network connection might cause the stream to stutter, freeze, and stall - but as the connection gets better the video is recovered quite well automatically.
\nOne of the complications presented by Apple's guidelines is that an application that streams must provide an audio only low-bitrate stream to compliment the higher bitrate video/audio streams. The AVPlayer framework does a great job doing what it needs to do switching the user to the best available stream. Unfortunately, if bandwidth gets really bad, the audio only stream is a valid option - and iOS WILL switch to it.
\nIt presents an interesting conundrum. Video looks broken to the user, and especially your clients who are footing the bill. I'd almost rather have the video stammer and stutter than to switch to this audio only bitrate - at least then, most tech savvy users would know that they are having network connectivity issues.
\nThe other problem is the ability to seek to exact positions in a stream using the AVPlayer in anything less than iOS 5. A stream is broken up into chunks. Those chunks are transport segment (.ts) files. Typically those chunks will be around 7-10 seconds each. Turns out that if you want to seek to a specific second of your stream, it better lie at the beginning of a segment, otherwise your seek accuracy is (at worst) 7-10 seconds off.
\nPast that, streams are pretty decent, like I said. What got frustrating though, was simple progressive video!
\nProgressive is fairly easy to play. You load, play and go. After loading, you need to wait for the item ready status, and then tell the content to play. By nature, progressive loads welllll....progressively. You can start playing back the video before the entire thing finishes loading.
\nThe limited API of AVPlayer seems to present some problems here. Your video is only "ready" once. What happens if you are progressively playing, but suddenly your connection drops low, and you can't quite finish playing the content because you've reached the end of whats been playing? I'll tell you what happens....your video pauses. While paused, the rest of the video will finish loading. Unfortunately, there is no status message to inform you that the video is ready to play again. So there it sits. Paused.
\nDrastic shortcomings call for stupid hacks. Such as setting up a timer to fire off every few seconds and tell the content to play. If the content is currently playing, the call to play will do nothing. However, if it's stalled and paused, we can keep the video moving along automatically.
\nAnother shortcoming I ran across is an unfortunate reality with the AVQueuePlayer. The AVQueuePlayer is like the AVPlayer, but you can playlist things. I chose to use this type of player due to other requirements in my project. For my work, I wasn't really using it as a queue. Instead, I'd simply replace the current item in the queue to play a new asset. Unfortunately this method of doing things didn't seem very reliable. If an item failed in my queue, I could add further items and make them play, however I wouldn't get any status events about the item. That means I'd never get told when it was ready to play, when it completed, when it failed. It was really a showstopper. Sometime, even if the item didn't fail to play, I'd see that subsequent items wouldn't produce status changes.
\nI ended up having to destroy the player and starting fresh each time - oh well. Even though I wasn't using it as a queue before, now I REALLY wasn't using it as a queue.
\nSo there it is. I got through it, but video was a little cranky!
\n", "date_published": "2012-01-28T00:00:00Z" } , { "id": "https://benfarrell.com/blog/2011-12-17-the-adsr-envelope-with-audiolib-js/", "url": "https://benfarrell.com/blog/2011-12-17-the-adsr-envelope-with-audiolib-js/", "title": "The ADSR Envelope with Audiolib.js", "content_html": "So, let's talk envelopes. First what they are, and what they can do to the quality of a sound.
\nDemo here: /labs/examples/envelopes-12-17/, but read on for how it all works!
\nAn envelope is a real simple concept actually. Take any signal, like a sound. Maybe that sound plays at a constant volume. When you apply an "envelope" to that sound, you are changing the volume of that sound while it's played. It might go up and down, back up again, whatever.
\nHow it goes up and down, and the speed at which the volume is changed is up to the details of the envelope used.
\nYou could create an envelope that takes 8 hours to complete. Maybe you want to go to sleep with some music, and then wake up in the morning with music. If you know it takes you 30 minutes to fall asleep, you'll start the music playing at a loud volume. Over the next 30 minutes, you envelope your sound from loud to quiet, to off. In the morning, 30 minutes before you wake up, the envelope makes the music go from off, to quiet, to loud again. It wakes you up!
\nWhile this 8 hours illustrates how an envelope works, it's a bit different when talking musical tones.
\nIt's not that different, however. We're still talking about volume over time, but we're talking milliseconds instead of hours or even minutes or seconds.
\nThe character or personality of a musical tone can be changed greatly by altering the envelope. While we talk about this in terms of 1/1000th of a second, you don't really notice the volume changing as you listen to the tone. Your ear doesn't necessarily detect that the volume is going up and down.
\nInstead, the sound just has a different tonal quality! A piano for example wouldn't sound as sharp if it didn't go from no sound to loud that quickly. An accordion though, has a longer time as it goes from quiet to loud. And that quality - the "attack" (amongst other factors) create the personality of the tone.
\nLet's talk specifics now. ADSR.
\nThat's Attack, Decay, Sustain, Release. The ADSR envelope is just one type of envelope, but it's a popular one that's been used in electronic music for decades.
\n![\"M4C_intro.ADSR\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2011/12/M4C_intro.ADSR_.png\")
\nthe attack, decay, sustain, and release phases
I've talked about Audiolib.js in previous posts. Audiolib is the Javascript library that enables you to make these dynamic sounds in Chrome and Firefox.
\nAudio programming isn't easy though! So while Audiolib helps out in awesome ways, it doesn't have concepts of notes and music theory. You have to tell it what frequencies to play, and if playing a chord, which individual frequencies make up the chord - which I explored and created my own helpers for.
\nThe ADSR envelope is another example of something that Audiolib.js provides, however, it doesn't provide any obvious usage for it.
\nThere's actually a VERY good reason for this. While, we're talking about envelopes on musical tones, the 8 hour sleepy-time envelope is another good usage example. And that's restricting envelopes to the volume of a sound. There are tons more examples in audio synthesis that envelopes can be applied. Like effects - you can apply a distortion effect to a sound (like a rock guitar). But you can envelope the amount of distortion which is applied to the guitar. This has nothing to do with volume, and everything to do with just how much of something is applied to something else.
\nSo, I'd like to create a usage of our ADSR envelope that is limited to producing a musical tone - especially in the example of producing live sound by using a trigger (here it will be your computer/laptop keyboard).
\nLet's start with our previous example where we extend the Audiolib.js Oscillator via a plugin. We extended it to simply take a musical notation, like an "A" and set the correct frequency:
\naudioLib.generators('Note', function (sampleRate, notation, octave){ // extend Oscillator for ( var prop in audioLib.generators.Oscillator.prototype) { this[prop] = audioLib.generators.Oscillator.prototype[prop]; } // do constructor routine for Note and Oscillator var that = this; // are we defining the octave separately? If so add it if (octave) { notation += octave; } that.frequency = Note.getFrequencyForNotation(notation); that.waveTable = new Float32Array(1); that.sampleRate = sampleRate; that.waveShapes = that.waveShapes.slice(0); }, {});
So let's figure out how to work in an ADSR envelope. The usage for the Audiolib.js version is like so:
\nmyEnvelope = audioLib.ADSREnvelope(sampleRate, attack, decay, sustain, release, sustainTime, releaseTime);
\nThe parameters work like so:
\nThe envelope has 6 states of being (0-indexed inside the code):
\nTo kick off our envelope, you trigger it:
\nmyEnvelope.triggerGate(true);
\nNow we can start using our envelope. The usage is a little weird to me, as the Audiolib.js library treats it like a "generator" which seems a bit complicated for what it does. I just want a stream of numbers, but OK I'll bite. I'll use it with the byte arrays and whatnot, as if it's an Oscillator.
\nvar buffer = new Float32Array(1); myEnvelope.append(buffer, 1);
\nSo, I'm just pulling one value at a time from the envelope, and putting it into my "buffer". But my buffer only has one value in it at any time. Like I said, I feel like I'm being forced into using it in more complicated of a way than I need! Maybe there's something I'm missing.
\nNow, every time, I get create an audio data point in my sound, I can multiply the data point by my envelope. Thus my envelope is applied!
\nthis[this.waveShape]() * buffer[0];
\nTo do this, I overrode the "getMix" function in the Audiolib Oscillator. But I needed to do other things too.
\nSince I trigger the envelope with triggerGate, it will cycle through the attack and release to the sustain phase as the envelope is used, automatically.
\nIt gets complicated at the release phase though. After your computer/laptop keyboard is released, we need to enter the release phase. But we're still grabbing sound, because the release phase still produces sound as it fades. So our Oscillator needs to track that it's in the release phase (it knows by keeping track of the envelope.state, which is 3 or 5 here for release or timed release).
\nThen finally when it gets back to state 0, or the cycle begins again, we mark this note as "released", so our buffer knows that it doesn't need to pull from it anymore. We have to be very careful of note pulling from more notes than we need, cause all this music stuff is hard work, and too many notes slows down your CPU and breaks the audio processing.
\nThe above is if the sustain phase goes on as long as you hold the key! What if it's a timed sustain - then we need the logic in there to release the key when the envelope is done, rather than when our user releases the key.
\nHere's our final Note.js code. And here's a controller for keeping track of keys being pressed.
\nIt all comes together in my (Chrome only) demo:
\n/labs/examples/envelopes-12-17/
\nThe demo starts out by not using an envelope at all. You'll hear some clicky-ness when you press and release a key. That's because you're hearing the transition between no sound and the abrupt start in the phase of the waveform. It's EXACTLY one of the reasons why envelopes are useful - to ease these transitions in and out.
\nWhen you turn on the envelope, you can start adjusting parameters to see how the different properties of attack, decay, sustain, and release affect the overall personality of the tone.
\n", "date_published": "2011-12-17T00:00:00Z" } , { "id": "https://benfarrell.com/blog/2011-12-13-chords-and-arpeggiators-with-audiolib-js/", "url": "https://benfarrell.com/blog/2011-12-13-chords-and-arpeggiators-with-audiolib-js/", "title": "Chords and Arpeggiators with Audiolib.js", "content_html": "So, as you may have noticed with my recent flurry of blog posts I've been playing with Audiolib.js. I can't thank the author Jussi Kalliokoski enough for getting this project going. When I did my Flash version, I spent many moons getting the audio buffer working correctly, then the oscillators, and I hadn't even gotten started on effects so much yet.
\nBut audiolib.js has all of this, which means I can focus on the usage with the API rather than making the low level audio stuff work.
\nFor example, Chords and Arpeggiation! Sorry! This only works in the latest version of Chrome
\nAfter a couple nights with the project, I extended out the standard Oscillator with a Note. I've since refactored a bit to do the minimum amount of music theory in the Note Oscillator object itself, and offload into a different object - but the net effect is that instead of instantiating a new Oscillator with a frequency, we can instantiate a new Note with a musical notation. For example with Oscillator, we'd say 440hz, with Note, we'd say "A" or "A4" (if you don't want to assume the 4th octave).
\nThe obvious extension to this is groups of notes - or rather Chords. There are two challenges here. The first is to convert a chord notation to array of notes. For example, if I say "Cm", or C Minor, I want an array such as ["C", "Eb", "G"]. We need to convert the C Minor notation into the implied triad of 3 notes using the minor scale. Also possible is a 6th, 7th, 9th, 11th, or 13th. In fact, now that I type, I realize I never did a 5th, which if I recall correctly, omits the middle note of a triad.
\nIt all came out fairly well. I don't think augmentation, sustain, or diminish work properly yet, but the chord structures are there. The simplest way to use this is to use ChordFactory instead of Chord. Simply call ChordFactory.createNotations("C", 4) to get a list of notes (strings) or ChordFactory.createNotes("C", 4) to get an array of Note Oscillators.
\nChordConstants = { MAJOR_TRIAD: "maj", MINOR_TRIAD: "m", SEVENTH: "7", MINOR_SEVENTH: "m7", MAJOR_SEVENTH: "maj7", NINTH: "9", MINOR_NINTH: "m9", MAJOR_NINTH: "maj9", ELEVENTH: "11", THIRTEENTH: "13", SIXTH: "6", MINOR_SIXTH: "m6", SUSTAIN: "sus", AUGMENTED: "aug", DIMINISHED: "dim" }; ChordFactory = { /** * create a list of notations from chord * @param chord notation * @param notation array (individual notes) */ createNotations: function createNotations(notation, octave) { var chord = new Chord(notation, octave); return chord.getNotations(); }, /** * create an array of note oscillators using the audiolib framework * @param sampleRate * @param notation */ createNotes: function createNotes(sampleRate, notation) { var chord = new Chord(notation); nts = chord.getNotations(); var osc; var oscs = []; for (var nt in nts) { osc = audioLib.generators.Note(sampleRate, nts[nt]) oscs.push(osc); } return oscs; } } function Chord(notation, octave) { var that = this; /** root note of chord */ that._root = "C"; /** octave of root */ if (octave) { that._rootOctave = octave; } else { that._rootOctave = null; } /** chord notation */ that._notation = notation ? notation : "Cmaj"; /** notes in built chord */ that._notes = []; /** * get notes from built chords * * @return notes */ this.getNotations = function() { return this._notes; } /** * chord notation setter * * @param notation */ this.setNotation = function(value) { this._notation = value; this.buildChord(); } /** * chord notation getter * * @return notation */ this.getNotation = function() { return this._notation; } /** * root note setter * * @param root */ this.setRoot = function(value) { this._root = value; this.buildChord(); } /** * root note getter * * @return root note */ this.getRoot = function() { return this._root; } /** * root octave setter * * @param octave */ this.setRootOctave = function(value) { this._rootOctave = value; this.buildChord(); } /** * root octave getter * * @return root octave */ this.getRootOctave = function() { return this._rootOctave; } /** * get notes in major triad * * @param root note * @param root octave * @return notes */ this.majorTriad = function(root, rootOctave) { return this.getStandardNotesInChordMakeup(root, true, false, rootOctave).slice(0,3); } /** * get notes in minor triad * * @param root note * @param root octave * @return notes */ this.minorTriad = function(root, rootOctave) { return this.getStandardNotesInChordMakeup(root, false, false, rootOctave).slice(0,3); } /** * get notes in seventh chord * * @param root note * @param root octave * @return notes */ this.seventh = function(root, rootOctave) { return this.getStandardNotesInChordMakeup(root, true, false, rootOctave).slice(0,4); } /** * get notes in major seventh chord * * @param root note * @param root octave * @return notes */ this.majorSeventh = function(root, rootOctave) { return this.getStandardNotesInChordMakeup(root, true, false, rootOctave).slice(0,4); } /** * get notes in minor seventh chord * * @param root note * @param root octave * @return notes */ this.minorSeventh = function(root, rootOctave) { return this.getStandardNotesInChordMakeup(root, false, false, rootOctave).slice(0,4); } /** * get notes in ninth chord * * @param root note * @param root octave * @return notes */ this.ninth = function(root, rootOctave) { return this.getStandardNotesInChordMakeup(root, true, false, rootOctave).slice(0,5); } /** * get notes in major ninth chord * * @param root note * @param root octave * @return notes */ this.majorNinth = function(root, rootOctave) { return this.getStandardNotesInChordMakeup(root, true, false, rootOctave).slice(0,5); } /** * get notes in minor ninth chord * * @param root note * @param root octave * @return notes */ this.minorNinth = function(root, rootOctave) { return this.getStandardNotesInChordMakeup(root, false, false, rootOctave).slice(0,5); } /** * get notes in eleventh chord * * @param root note * @param root octave * @return notes */ this.eleventh = function(root, rootOctave) { return this.getStandardNotesInChordMakeup(root, true, false, rootOctave).slice(0,6); } /** * get notes in major eleventh chord * * @param root note * @param root octave * @return notes */ this.majorEleventh = function(root, rootOctave) { return this.getStandardNotesInChordMakeup(root, true, false, rootOctave).slice(0,6); } /** * get notes in minor eleventh chord * * @param root note * @param root octave * @return notes */ this.minorEleventh = function(root, rootOctave) { return this.getStandardNotesInChordMakeup(root, false, false, rootOctave).slice(0,6); } /** * get notes in thirteenth chord * * @param root note * @param root octave * @return notes */ this.thirteenth = function(root, rootOctave) { return this.getStandardNotesInChordMakeup(root, true, false, rootOctave).slice(0,7); } /** * get notes in major thirteenth chord * * @param root note * @param root octave * @return notes */ this.majorThirteenth = function(root, rootOctave) { return this.getStandardNotesInChordMakeup(root, true, false, rootOctave).slice(0,7); } /** * get notes in minor thirteenth chord * * @param root note * @param root octave * @return notes */ this.minorThirteenth= function(root, rootOctave) { return this.getStandardNotesInChordMakeup(root, false, false, rootOctave).slice(0,7); } /** * get notes in sixth chord * * @param root note * @param root octave * @return notes */ this.sixth = function(root, rootOctave) { var keySig = Note.notesInKeySignature(root, true, rootOctave); var keys = new Array(); keys.push(keySig[0], keySig[2], keySig[4], keySig[5]); return keys; } /** * get notes in minor sixth chord * * @param root note * @param root octave * @return notes */ this.minorSixth = function(root, rootOctave) { var keySig = Note.notesInKeySignature(root, false, rootOctave); var keys = new Array(); keys.push(keySig[0], keySig[2], keySig[4], keySig[5]); return keys; } /** * sustain chord * * @param notes * @param direction to sustain * @return notes */ this.sustain = function(notes, sus) { sus = (sus == undefined) ? 4 : sus; // grab the third in the chord var third = notes[1]; var notations = Note.sharpNotations; var thirdIndex = Note.sharpNotations.indexOf(third); if (thirdIndex == -1) { notations = Note.flatNotations; thirdIndex = Note.flatNotations.indexOf(third); } if (sus==2) { // lower the third one half step if (thirdIndex-1 = notations.length) { notes[1] = notations[0]; } else { notes[1] = notations[thirdIndex+1]; } } return notes; } /** * augment chord * * @param notes * @return notes */ this.augment = function(notes) { // grab the fifth in the chord var fifth = notes[2]; var notations = Note.sharpNotations; var fifthIndex = Note.sharpNotations.indexOf(fifth); if (fifthIndex == -1) { notations = Note.flatNotations; fifthIndex = Note.flatNotations.indexOf(fifth); } // raise the fifth one half step if (fifthIndex+1 >= notations.length) { notes[2] = notations[0]; } else { notes[2] = notations[fifthIndex+1]; } return notes; } /** * get all standard notes in a chord, from triad to thirteenth * * @param root note * @param major key (true/false) * @param major chord (true/false) * @param root octave * @return notes array */ this.getStandardNotesInChordMakeup = function(root, majorKey, majorChord, octave) { majorKey = (majorKey == undefined) ? true : majorKey; majorChord = (majorChord == undefined) ? false : majorChord; var majKeySig = Note.notesInKeySignature(root, true, octave); var minKeySig = Note.notesInKeySignature(root, false, octave); // grab the next octave if we need it var majKeySig2 = Note.notesInKeySignature(root, true, octave+1); var minKeySig2 = Note.notesInKeySignature(root, false, octave+1); var notes; if (majorKey && majorChord) { // C Major Seventh for example notes = [ majKeySig[0], majKeySig[2], majKeySig[4], majKeySig[6], majKeySig2[1], majKeySig2[3] ]; } else if (!majorKey && majorChord) { // C Minor Seventh for example notes = [ minKeySig[0], minKeySig[2], minKeySig[4], minKeySig[6], minKeySig2[1], minKeySig2[3] ]; } else if (majorKey && !majorChord) { // C Seventh for example notes = [ majKeySig[0], majKeySig[2], majKeySig[4], minKeySig[6], majKeySig2[1], minKeySig2[3] ]; } else if (!majorKey && !majorChord) { // C Seventh for example notes = [ majKeySig[0], minKeySig[2], majKeySig[4], minKeySig[6], majKeySig2[1], minKeySig2[3] ]; } return notes; } /** * convert notation to note list * * @param notation * @param use the octave in the notation * @return note list */ this.notesFromChordNotation = function(notation, octave) { var root; var major = 0; var chordType; // find root if (notation.charAt(1) == "#" || notation.charAt(1) == "b") { root = notation.substring(0, 2); notation = notation.substring(2); } else { root = notation.substring(0, 1); notation = notation.substring(1); } // major or minor? (3 states - 1 is on, -1 is off, 0 is unspecified) if ( notation.substr(0, 3) == "maj") { major = 1; notation = notation.substring(3); } else if (notation.substr(0, 1) == "m") { major = -1; notation = notation.substring(1); } // set chord type if ( notation.charAt(0) == "6" ) { if (major == -1) { chordType = ChordConstants.MINOR_SIXTH; } else { chordType = ChordConstants.SIXTH; } notation = notation.substring(2); } else if ( notation.charAt(0) == "7" ) { if (major == 0) { chordType = ChordConstants.SEVENTH; } else if (major == 1) { chordType = ChordConstants.MAJOR_SEVENTH; } else if (major == -1) { chordType = ChordConstants.MINOR_SEVENTH; } notation = notation.substring(1); } else if ( notation.charAt(0) == "9" ) { if (major == 0) { chordType = ChordConstants.NINTH; } else if (major == 1) { chordType = ChordConstants.MAJOR_NINTH; } else if (major == -1) { chordType = ChordConstants.MINOR_NINTH; } notation = notation.substring(1); } else if ( notation.substr(0,2) == "11" ) { chordType = ChordConstants.ELEVENTH; notation = notation.substring(2); } else if ( notation.substr(0,2) == "13" ) { chordType = ChordConstants.THIRTEENTH; notation = notation.substring(2); } else { if (major == 1 || major == 0) { chordType = ChordConstants.MAJOR_TRIAD; } else { chordType = ChordConstants.MINOR_TRIAD; } } var notes = this.notesFromChordType(chordType, root, octave); // modify note set if needed var modifier = notation; switch ( modifier.substr(0,3) ) { case ChordConstants.AUGMENTED: notes = augment(notes); break; case ChordConstants.DIMINISHED: // to do break; case ChordConstants.SUSTAIN: var param = int(modifier.charAt(3)); notes = sustain(notes, param); break; } return notes; } /** * get notes from chord types * * @param type * @param chord root * @return notes */ this.notesFromChordType = function(type, root, rootOctave) { switch ( type ) { case ChordConstants.SIXTH: return this.sixth(root, rootOctave); case ChordConstants.MINOR_SIXTH: return this.minorSixth(root, rootOctave); case ChordConstants.SEVENTH: return this.seventh(root, rootOctave); case ChordConstants.MINOR_SEVENTH: return this.minorSeventh(root, rootOctave); case ChordConstants.MAJOR_SEVENTH: return this.majorSeventh(root, rootOctave); case ChordConstants.NINTH: return this.ninth(root, rootOctave); case ChordConstants.MINOR_NINTH: return this.minorNinth(root, rootOctave); case ChordConstants.MAJOR_NINTH: return this.majorNinth(root, rootOctave); case ChordConstants.ELEVENTH: return this.eleventh(root, rootOctave); case ChordConstants.THIRTEENTH: return this.thirteenth(root, rootOctave); case ChordConstants.MAJOR_TRIAD: return this.majorTriad(root, rootOctave); case ChordConstants.MINOR_TRIAD: return this.minorTriad(root, rootOctave); default: return this.majorTriad(root, rootOctave); } } /** * build the chord given the parameters set in this class */ this.buildChord = function() { this._notes = []; var notations = this.notesFromChordNotation(this._notation, this._rootOctave); for (var c = 0; c < notations.length; c++) { this._notes.push(notations[c]); } } // do a build based on initial params that.buildChord(); }
Namespacing will probably come when I think of a name for this project!
\nAnyway, we have some code that creates chord structures! Yay! Now, what about sending this to the buffer for mixing. The most basic example in audiolib.js is like so:
\nfunction audioCallback(buffer, channelCount){ // Fill the buffer with the oscillator output. osc.append(buffer, channelCount); }
This takes the Oscillator (or any audio generator for that matter) and generates some bytes to send to the audio buffer. Audiolib had some examples of mixing, but I didn't care for them so much, as they seemed a little overly complicated with whatever idea of "leads" they had - I was a bit confused. In the end, I came up with something that worked a little better for me, but is probably virtually the same concept:
\nfunction audioCallback(buffer, channelCount){ // get a list of generators from our keyboard controller object - it keeps tracks of keys pressed var gens = ctrl.pull(); var bl = buffer.length; var gl = gens.length; // loop through each sample in the buffer for (current=0; current<bl; current+= channelCount){ sample = 0; // here we're combining samples from our list of generators // and combining down into one sample for (i=0; i<gl; i++){ gens[i].generate(); sample += gens[i].getMix()*0.5; } // Here we write this sample to each channel we have (I'm just doing 2 channels) for (n=0; n<channelCount; n++){ buffer[current + n] = sample; } } }
That's pretty much it! We're constructing Oscillators from Chord notations, and then mixing the result for playback.
\nThe last thing I wanted to try was an arpeggiator. An arpeggiator takes a chord structure, and instead of playing all the notes at the same time, only plays one note at a time in sequence.
\nI extended the Audiolib Oscillator again, making an Arpeggiator plugin. I needed to override the "generate" method. Every time "generate" is called, I'd step through. Once my step count reaches a threshold (the samplerate divided by some number), I'd change to the next frequency.
\nHere's what I got:
\naudioLib.generators('Arpeggiator', function (sampleRate, arpeggiatorRate, notes, octave, autoReverse){ // extend Oscillator for ( var prop in audioLib.generators.Oscillator.prototype) { this[prop] = audioLib.generators.Oscillator.prototype[prop]; } // do constructor routine for Note and Oscillator var that = this; if (autoReverse === undefined) { that.autoReverse = true; } else { that.autoReverse = autoReverse; } that.buildFrequencies(notes, octave); that.waveTable = new Float32Array(1); that.sampleRate = sampleRate; that.waveShapes = that.waveShapes.slice(0); that.arpStep = 0; that.arpIndex = 0; that.arpeggiatorRate = arpeggiatorRate; /** * override generate function */ that.generate = function(){ var self = this, f = +self.frequency, pw = self.pulseWidth, p = self.phase; f += f * self.fm; self.phase = (p + f / self.sampleRate / 2) % 1; p = (self.phase + self.phaseOffset) % 1; self._p = p self.sampleRate * self.arpeggiatorRate) { self.arpStep = 0; self.arpIndex ++; if (self.arpIndex >= self.frequencies.length) { self.arpIndex = 0; } self.frequency = self.frequencies[self.arpIndex]; } } }, { buildFrequencies: function(notes, octave) { var self = this; self.frequencies = []; if (!(notes instanceof Array)) { // work with either a chord structure... notes = ChordFactory.createNotations(notes, octave); } for (var c = 0; c 0; c--) { self.frequencies.push( Note.getFrequencyForNotation(notes[c]) ) } } self.frequency = self.frequencies[0]; } });
Lotsa code here tonight! I'd like to get started on some basic effects and envelopes next to make some non-boring tonal qualities to my playback. At that point, I hope to have a nice little useable API I can put up somewhere - especially since I don't have to put lotsa code up!
\n", "date_published": "2011-12-13T00:00:00Z" } , { "id": "https://benfarrell.com/blog/2011-12-05-making-an-audiolib-js-plugin/", "url": "https://benfarrell.com/blog/2011-12-05-making-an-audiolib-js-plugin/", "title": "Making an Audiolib.js Plugin", "content_html": "I just started checking out Audiolib.js this weekend. Audiolib is a Javascript project to help you synthesize sounds and tones in JS.
\nJust recently, Google Chrome added the Web Audio API, while Firefox added the Audio Data API. Both let you get low level with sound - you can add bytes to an audio buffer and manufacture sound in realtime.
\nThe library is bundled with "sink.js" which handles the inconsistencies in the audio API between Firefox and Chrome to create a common API. Once this is abstracted away, we can make some audio.
\nI've done the nitty gritty of writing bytes to the buffer in Flash, and from my limited playing Audiolib does a great job with this, and I'm ecstatic I don't have to rewrite my Flash stuff!
\nAudiolib provides "generators" for you to work with. A "generator" is something that makes a sound, and defines a common API for anything that makes a noise to write that sound to the audio buffer.
\nIn the generators namespace/package, we have an Oscillator, White/Pink/Brown Noise, and a Sampler. White noise is cool and all, but I jumped right to the Oscillator to make a real tone. The Oscillator takes a sample rate and a frequency.
\nYou can define it thusly: dev = audioLib.AudioDevice(audioCallback /* callback for the buffer fills */, 2 /* channelCount */); osc = audioLib.Oscillator(dev.sampleRate /* sampleRate */, 440 /* frequency */);
So, first we defined the device to use - we told it what to use for the audio callback, and how many channels there are (we have 2 channels...left and right).
\nThe audio callback is a pretty standard thing in the world of audio. When the sound card is starting to run out of audio to play, it signals Javascript and says "Hey I need more audio! Fill me up with some bytes". With our "audioCallback" function, we say "No problem sound card! I got your back! When you run low, please call our audio callback method - this will fill your audio buffer up"
\nHere's an example of the audio callback: function audioCallback(buffer, channelCount){ osc.append(buffer, channelCount); }
All that happens here, is that the buffer and channel count gets passed into the method, and we tell out Oscillator what this is, and this Oscillator generates the appropriate bytes and send them to the buffer.
\nBut what is that generator...the Oscillator? Well, it creates a sound at a certain frequency. We pass in the frequency as we instantiate this "osc" object. When the audioCallback fires, it pulls this frequency from the osc object and plays a specific tone!
\nThis is awesome! All the hard work is done for us - but one of the first things I wanted to tackle here was to make it easier for musicians to understand.
\nSee, casual musicians don't know that a middle "A" on a piano oscillates at 440hz - they just know that they hit a middle "A" to play the tone. I want something that makes sense for casual musicians, so my first small task with audiolib is to override Oscillator to take a notation vs a frequency.
\nAudiolib provides a plugin spec, but it tripped me up a little to understand how a "Generator" worked. See, they have a js/generation folder which contains the Oscillator. It's all quite readable and self-documented.
\nHowever, the "append" method was not found in the Oscillator class at all! I was looking EVERYWHERE for it!
\nEventually I found it in the "wrapper-end.js" script that's outside of all the folders.
\nBasically, you create a generator like Oscillator, and you write it to the audiolib.generators namespace. By virtue of being in this namespace, the "wrapper-end" script comes along to wrap things up. It takes all the things in the audiolib.generators namespace and adds the generator base functions. Basically it makes all the generators extend the generator base class after the fact.
\nKinda confusing and sneaky if you ask me! Oh well.
\nRegardless, the plugins are well documented, so I copied the sample generator plugin:
\naudioLib.generators('SemiClock', function (sampleRate){ this.sampleRate = sampleRate; }, { prevSample: 0.0, generate: function(sample){ this.phase = + !this.phase; }, getMix: function(){ return this.phase; }, phase: 0 });
Cool, so we're naming a generator in the generator namespace, and defining some required methods, like "generate" and "getMix".
\nWell, all I want to do is make a copy of all the objects in Oscillator and add a function and initialization routine to accept a musical notation and instantiate an oscillator with the correct frequency.
\nSo here's my take: audioLib.generators('Note', function (sampleRate, notation, octave){ // extend Oscillator for ( var prop in audioLib.generators.Oscillator.prototype) { this[prop] = audioLib.generators.Oscillator.prototype[prop]; } // do constructor routine for Note and Oscillator var self = this; self.octave = isNaN(octave) ? 4 : octave; self.setNotation(notation); self.waveTable = new Float32Array(1); self.sampleRate = sampleRate; self.waveShapes = self.waveShapes.slice(0); }, { /* incremental tones as sharp notation */ sharpNotations: ["A", "A#", "B", "C", "C#", "D", "D#", "E", "F", "F#", "G", "G#"], /* incremental tones as flat notation */ flatNotations: ["A", "Bb", "B", "C", "Db", "D", "Eb", "E", "F", "Gb", "G", "Ab"], /** * notation setter * @param notation (octave is optional) */ setNotation: function(nt) { this.notation = nt; // does notation include the octave? if ( !isNaN( parseInt(nt.charAt(nt.length -1)) )) { this.octave = parseInt(nt.charAt(nt.length -1)); this.notation = nt.substr(0, nt.length-2); } this.frequency = this._getFrequencyForNotation(this.notation); }, /** * turn a notation into a frequency * @param notation * @return frequency */ _getFrequencyForNotation: function(nt) { var freq; var indx = this.sharpNotations.indexOf(nt); if (indx == -1) { indx = this.flatNotations.indexOf(nt); } if (indx != -1) { indx += (this.octave-4) * this.sharpNotations.length; freq = 440 * (Math.pow(2, indx/12)); } return freq; } });
Basically what I did here is extended the Oscillator class. I looped through all the properties of the Oscillator.prototype object and copying it onto the new class.
\nI then, went in to the Oscillator generator and copied the small initialization routine.
\nThen I injected my own methods! I replaced the frequency setting with a method to lookup the index of the musical notation in my preset arrays of notations. I also parsed out the octave (if it existed).
\nIn the end, I have a "Note" class which is exactly the same as the Oscillator class - however, instead of passing in the frequency (ie 440hz), you pass in "A", or "A4" for the 4th octave.
\nTo change the notation of the oscillator/note, simply call the setNotation again!
\nI'm definitely looking forward to exploring Audiolib.js more. It has a few concepts that I hadn't gotten around to implementing in my own Flash framework yet, so I'm excited to see how it's done.
\n", "date_published": "2011-12-05T00:00:00Z" } , { "id": "https://benfarrell.com/blog/2011-12-02-javascript-we-dont-need-no-stinkin-blueprints/", "url": "https://benfarrell.com/blog/2011-12-02-javascript-we-dont-need-no-stinkin-blueprints/", "title": "Javascript: We don't need no stinkin Blueprints", "content_html": "So, at this point I feel like I'm a competent JS developer. Not pro, just competent. Since I can't just let things lie there, I'm reading "Secrets of the Javascript Ninja" and trying to get a handle on what Javascript, and even a functional language mean as you work with them.
\nOne of the biggest complaints I have so far is the openness that the language allows. It sounds stupid I know, because why wouldn't it be a good thing to allow you to do anything you want?
\nWell, I like to learn by doing. After I "do", and finish one or two projects, I start to question the underlying ways of doing things, and I learn.
\nI'm not the type to read about the building blocks for several weeks, and say "Hmmmmm.....I understand these building blocks, now is the time to code".
\nNope! I just plow through - it's OK I don't understand the fundamentals, if I'm following a best practice, then I can trust that the best practice is correct and learn as I go.
\nComing from Actionscript, I obviously want to be more object oriented. I've finally grasped that there's no good way to emulate this without something like John Resig's Class construct: http://ejohn.org/blog/simple-javascript-inheritance.
\nThat's fine, but it took me a while to accept this. After all, Javascript is pretty much object-oriented, why is it such a chore to make classes and extend them in an OO fashion?
\nI had to take a big step back to think about what languages like Java and AS3 do to make them what they are. The big difference, I found is NOT that Javascript lacks OO features, but that it lacks blueprints/definitions.
\nBlueprints are one of those things that have always been there for me, but I always took them for granted. Also known as definitions, they sit in around waiting to be spun up as an object in memory.
\nThese blueprints basically serve as a model for how, once you instantiate the object, what it will look like and how it will function after you call new MyBlueprint().
\nYou can let a blueprint know what properties and methods it's supposed to have. You can also say that BlueprintX extends BlueprintY (inheritance), and it implements the same type of methods as IBlueprint (an interface).
\nIt's all very convenient, and gives a good model of how to structure your code. It also sits in the background, lying in wait to be spun up. I'm assuming that it doesn't cost anything at runtime besides the memory to store the text of the class.
\nAnyway, it seems that Javascript doesn't have the notion of a blueprint! The only thing close to a blueprint is the base Object. Apparently, the only blueprint in Javscript is stored in the Object.prototype property. You can't define any thing extra before your browser instantiates! Oh well. It's just something to accept and move on, but it really was a major mental block for me in how the language functions.
\nInstead, when you need to create a blueprint like thing, you create whatever you need at runtime. Javascript allows you to create a base object, set whatever properties you want on it, and then manually clone it to other types of things.
\nSo you can certainly create something that you'd like to use as a blueprint, but you are instantiating this model at runtime as a normal everyday object. It resides in memory like any other object. It's just that YOU, the programmer decides to use it as a blueprint - so you make the decision to clone it, extend it, mix it with other inheritances, etc. All manually.
\nNo big deal, but like I said, the lack of a formal blueprint model was a real head scratcher for me, and it made me think about what I was missing and was taking for granted in every other language I've ever used! My initial knee jerk reaction was to say "Javascript isn't OOP", but that's not true - it just doesn't need any stinkin blueprints.
\n", "date_published": "2011-12-02T00:00:00Z" } , { "id": "https://benfarrell.com/blog/2011-11-15-thoughts-er-wild-unsubstantiated-predictions-on-flash/", "url": "https://benfarrell.com/blog/2011-11-15-thoughts-er-wild-unsubstantiated-predictions-on-flash/", "title": "Thoughts....er...Wild Unsubstantiated Predictions on Flash", "content_html": "Yes, Adobe did us, the developers wrong this week. No doubt about it. Even if they have the best possible intentions, they did an AWESOME job salting the developer community earth this week.
\nIt got me thinking, assuming they did have the best possible intentions for Flash, what could that mean? What is going on behind the closed doors and tight lips?
\nOK, let's revisit what they said: Adobe is Stopping development on Flash Player for browsers on mobile.
\nRead it again...it says Adobe is stopping development. They already have a partner developer program, which means that a company can come in, sign an NDA, and make their own Flash player.
\nThey ALSO have the Open Screen Project. The project consists of a lot of companies all pledging to support the SWF format reliable on many screens.
\nMy knee jerk reaction to this past week's news is "Hoooooo boy.....HTC, Google, MTV, Disney, Cisco, Samsung, etc, etc, are all gonna be super pissed that Adobe is dropping Flash and essentially this project".
\nWhat if this news revolves around the OpenScreen project? We've heard from some of the senior Flash engineers how much of a pain it is to maintain Flash on every device in every browser. There are way too many screens. The message we're hearing from the Flash engineers is that they will still continue desktop development.
\nMy thought process leads to Adobe's PDF history here. Adobe created the spec, and owned Acrobat for many years. The spec was released in 2001 and was released as an open standard in 2008. This means that anyone can create tools that make or read PDFs. We have Foxit, Google Chrome Support, OSX support, the list goes on. Adobe still has Acrobat, and while you may think it's bloated and you complain about it non-stop, its still a very popular method to read and write documents. We have other text document standards as well of course...like HTML!
\nDoes this sound like another open specification we know? Yep the Flash SWF specification. The spec is open, but the player is not open source. Adobe has told us in the past that proprietary tech like the h264 video format prevents them from releasing the code in it's entirety. However, we're now at a cross-roads with Flash. Lots of stuff available on the desktop is not available on a device (3D) and vice versa (gestures, acceleromter, GPS, etc). Flash is no longer consistent everywhere.
\nThis feels to me like a good opportunity for Adobe to maintain it's vision for Flash on the desktop and say "Guys, if you are implementing Flash on your screen...this is what you should shoot for". But vendors like Sony can also say, "Wellllll, the 3D stuff won't work on my cheap TV, we're just gonna take the other stuff". It also feels like a good stopping point for them if they were to hand this off to the OpenScreen foundation. They have massive penetration on the desktop, and a decent inroads on mobile devices.
\nWhere Adobe leaves off, Android/Google can pick up for their own OS - and do it BETTER than Adobe because they know their own devices better than Adobe ever could.
\nAdobe could certainly leave h264 video out of the open sourced player, and leave it up to vendors to pay the licensing fee and hook it into their version of Flash.
\nAlso, go back a few months, we saw AIR 3 announced with native extensions, fast native performance. All of this is designed to make SWFs a powerful format. It also makes a path for vendors to hook into Flash in whatever way they want.
\nAnyway - I like working with Flash. Just like I like working with a lotta cool tech. This is just my optomistic vision of what is going on behind closed doors, and its probably not even true. Even if it was true, Adobe lost a lot of trust with developers. But I'm not here to whine, that's what the rest of the internet is for!
\nBut, I'll leave you with this...if it was being turned over to the OpenScreen project and managed by many companies, how much work and wrangling would it take IN SECRET to make all of these companies happy before announcing intentions.
\nBuy into this or not, I don't care! Just a wild theory.
\n", "date_published": "2011-11-15T00:00:00Z" } , { "id": "https://benfarrell.com/blog/2011-10-12-c-web-run-run-web-run/", "url": "https://benfarrell.com/blog/2011-10-12-c-web-run-run-web-run/", "title": "C Web Run, Run Web Run", "content_html": "After a long vacation, i'm finally back from Adobe MAX and California. Going to MAX certainly gave a lot of perspective on things.
\nAdobe's keynote around Flash and HTML5 seemed to be a hot button for some people. Flex wasn't mentioned, and Flash was named "the console of the web". This seemed to draw ire from people who know Flash is more than games.
\nI actually think the verbage is just fine. Flash is the box in your living room that runs nifty stuff. It does applications, video, games, and all the cool stuff that is net connected but isn't a "web page".
\nObviously, Flash does RIA applications because it does everything else. It's a little unfortunate that RIAs aren't part of the overall message, but to be realistic, when normal people talk RIAs, they are thinking of what the HTML/JS/CSS stack can do easily---and if you want to claim that Flex should be used where HTML/JS/CSS can be easily used, you are fighting a losing battle these days.
\nFor the rest of us technically adept people, we know in which situations Flash can or can't be used. It's unfortunate it's been reduced to "web standards" versus "the console," but Adobe had to put it in terms the world could understand. I think they threw in the towel on a small battle to win the larger war to keep Flash going strong.
\nThis is all interesting, but it's not my takeaway from MAX. Adobe gives us a stack for the console: Flex/AIR/Flash. They also give us a stack for standards-based Web: JQuery/HTML/PhoneGap. The untold story taking shape in the creative space is low level and close to the metal, as they say. C++ is heading to the creative space and the Web. It is slowly taking shape as the technology that can be injected into the Web stack, the Flash stack, and the creative expression stack.
\nPrior to MAX, I had been looking into OpenFrameworks (OF). It's a C++ library that allows people to creatively express themselves using C++. This is significant because C++, in the past, was seen as one of those daunting languages that nobody wanted to touch because it was too complicated and reserved for computer rocket scientists. But OF aims to make C++ approachable for creative types and allows them to push graphics around and do some cool things like face tracking, object detection, and more. Basically, anything creative that Flash or Javascript is too slow to handle, C++ can do and OF is your path to get there.
\nSure, OF is significant, but it's just part of the story. Adobe has another project in labs called Alchemy. They recently announced in a MAX session that it would be getting a lot more attention...official product support attention! You may remember Alchemy as a MAX demo a few years back. The source code for Quake was compiled from C++ through Alchemy and used as a Flash-compiled library. The end result was that a Quake level was cross compiled and playable in Flash (and this was before Stage3D).
\nAlchemy is used to compile existing C++ libraries or just code you'd like to run that goes too damn slow in Flash/Actionscript. This opens up a world of possibilities for near native performance. In fact, remember that Photoshop Touch demo you saw in the Day 1 keynote? That was built with AIR. Still, Actionscript is just a shade too slow to run Photoshop's advanced image manipulation algorithms. Instead, the developers simply leveraged Photoshop's existing C++ libraries and plugged them into AIR/Flash.
\nThat story was the same as the one I overheard from Ben Garney of PushButton Labs when talking about his game development work. He coded up a library in C++, which could be targeted at normal native usage. When needed in Flash, the library can be simply compiled with Alchemy for usage in Flash.
\nThe benefit is immense here---incredible performance boosts and the ability to leverage the same code everywhere that runs C++.
\nBut the Flash story doesn't stop there. AIR 3.0 allows native extensions on both Desktop or mobile applications. Whether you use C++ to create a windows DLL or use the NDK to create an Android extension, it doesn't matter. Objective-C for iOS might be a slighly different flavor...but Apple isn't known for playing nice.
\nThe benefits of Flash are huge, but what about beyond Flash?
\nGoogle Chrome is part of the same story with their "Native Client." Just recently, Google gave us the ability to leverage C++ code in their Web browser for more power and speed. Meanwhile, projects like Emscripten will compile C++ to Javascript. While Emscripten doesn't equate to a speed boost, you can leverage exisiting C++ libraries. This is important because C++ has been around for decades, and someone before you may have already solved your problem.
\nWhere is all of this going? Well, I conjecture that we will end up doing the meaty bits of our apps/applications/Webpages with C++. We'll share the same C++ libraries across Flash, AIR, Desktop, Mobile, and Web. We'll then layer a thin client on top whether or not we write it in Javascript, MXML, Actionscript, Dart, HTML, etc. This layer will be visual, with the bottom C++ layer used for perfomance. I may be completely wrong about this prediction. Even if I am, the current creative ways to express ourselves with C++ give us normal folk a definite inroad into learning the most popular language on the planet.
\nIn other words, if you take the opportunity now to learn C++, what could possibly go wrong?!
\n", "date_published": "2011-10-12T00:00:00Z" } , { "id": "https://benfarrell.com/blog/2011-09-15-live-instrumentation-in-flash-part-6-some-demos/", "url": "https://benfarrell.com/blog/2011-09-15-live-instrumentation-in-flash-part-6-some-demos/", "title": "Live Instrumentation in Flash Part 6 – Some Demos", "content_html": "Demos
\nHello from NCDevCon. If you were able to attend my presentations, thanks!
\nI've gone ahead and written them up as blog posts for you if you'd like to revisit what I talked about.
\nHTML5 vs Flash Video: Choose Wisely (slides)
\nLive Instrumentation in Flash (slides)
\n", "date_published": "2011-09-15T00:00:00Z" } , { "id": "https://benfarrell.com/blog/2011-09-09-html5-vs-flash-video-choose-wisely-part-9-the-verdict/", "url": "https://benfarrell.com/blog/2011-09-09-html5-vs-flash-video-choose-wisely-part-9-the-verdict/", "title": "HTML5 vs Flash Video: Choose Wisely Part 9 – The Verdict", "content_html": "So what wins? HTML5 video or Flash? What should you use, what's better?
\nThe answer to this is that it depends on your project and what platforms you are targeting.
\nUse HTML5 video for simple playback
\nFor short form video that contains no in-stream advertisements (that's fancy talk for pre, post, and midrolls) its extremely easy to use HTML5 now. If your content is at most a few minutes long and you want it to work on any browser and any device you should use HTML5 with a Flash fallback (or vice versa).
\nYour content should be MP4/h264 so that it will work on iOS, Flash, Chrome, and IE9. If you feel feisty, you can make a copy of your content as OGG to play in Firefox, but you can easily handle Firefox with Flash video.
\nUse Flash for in-depth Ad and Tracking support
\nAdding ads can be a little trickier. It's much easier to do this with Flash, but if your content is short, all you'll most likely need is a short pre-roll. If doing this with HTML5, the experience may be a bit clunky, but at least it will work on the iPad.
\nIt stinks a little more as you add tracking (or other features). Do you want to duplicate your programming efforts in both Flash and Javascript?
\nOne suggestion might be to implement a Flash player with no control bar and to only handle simple playback of video. Do all your graphics, reporting and ad logic in Javascript, and just use the Flash player in the same way you'd use a video tag. In this way, you're using two things that are pretty interchangable.
\nAll in all, you'll be most successful if you implement these features in Flash as Flash has the most support for this type of thing. HTML5 is certainly possible, but hard - you'll need to power through it, especially if you need it to work on iOS.
\nUse HTML5 for iOS If your only options are Flash or HTML5, obviously you should use HTML5 for playback since Flash isn't supported!
\nUse Flash for Long-Form Streaming Content
\nThis is a hard truth - HTML5 can't do streaming. As such, it can't do long-form content without completely bogging down end user's browsers.
\nFor playback on Android devices, you can use your same Flash player for long-form streaming playback (just make sure to make it mobile and touch friendly).
\nFor iOS, you're pretty much out of luck. You can run streaming long form content through iOS, but unfortunately that same content won't work on normal browsers. If you want playback on both desktop browsers and iOS devices, you'll need to double your efforts and develop players for each scenario.
\n", "date_published": "2011-09-09T00:00:00Z" } , { "id": "https://benfarrell.com/blog/2011-09-09-html5-vs-flash-video-choose-wisely-part-8-advertising-and-reporting/", "url": "https://benfarrell.com/blog/2011-09-09-html5-vs-flash-video-choose-wisely-part-8-advertising-and-reporting/", "title": "HTML5 vs Flash Video: Choose Wisely Part 8 – Advertising and Reporting", "content_html": "Before we conclude this series, I should mention something nobody likes - but they are an absolute necessity if content producers are going to put their video online in the first place.
\nThose things are ads and reporting.
\nContent owners want to monetize their video, and be able to track many aspects of it's usage to make better decisions about video you're watching.
\nWith Hulu, Netflix, and many networks putting first run television online it seems that web video is gaining on what we watch on cable TV, and we may see a day soon when the web is significant enough to eat into cable tv's revenue model such that we'll networks and content producers who want to thrive can't afford to just be on one or the other.
\nAs ad revenue from cable tv and network broadcasts are shifted, owners will want to make sure this money is JUST shifted and not lessened. So even today, folks are experimenting with longer ad breaks to see if they can get away with the same video experience you see on TV.
\nTo accomplish this, we need some good 3rd party libraries from ad companies and reporting companies, as well as decent support on the actual device, browser, or plugin to facilitate these advanced needs. If we can't come up with creative solutions to accommodate ads and reporting, companies will simply not come - and this can make or break a platform's usefulness for serving video.
\n3rd Party Library maturity
\nI've been working with Flash video extensively for a few years now, and in that time I've seen 3rd party frameworks and libraries delivered to us from various ad and reporting companies (many revisions in fact). Flash seems to have a lock on maturity of 3rd party libraries for handling these things. Flash has almost maniacal support for every aspect of playing video - something you won't see from HTML5, or even iOS (from what I've seen so far). In truth, this is why Flash is difficult to work with for video newcomers. But, at the same time it offers extreme control to allow almost anything we need. We can offer midrolls, prerolls, postrolls, companions, overlays and more. We can also track video buffering events, stream transitions, and many more minute details.
\nHTML5 video, on the other hand doesn't have this level of control - especially in iOS given the playback obstacles we've discussed. Given this - many 3rd party libraries are still figuring out how to offer the same level of ads and reporting we can obtain from Flash players.
\nOne company I work with just recently started offering midroll ads, where before their Javascript/HTML5 video solution wasn't mature enough to do this. They seem to be coming along just fine, but won't be caught up to Flash anytime soon! See part 6 for my experiences working with mid roll ads.
\nAdvertising with MP4 - At Least that's Easy!
\nI've talked about iOS streaming for long-form content. It uses Apple's HLS, and is pretty much incompatible with Flash or Silverlight. As a result, content owners will need to completely re-encode their streams for Apple devices - that is IF they want it to be streaming.
\nMany advertisers these days, however, will serve up their ad creatives as mp4 files encoded with h264. This works on Flash, and it ALSO works on iOS, Chrome, and IE. This means that if these advertisers wanted to use their same ads on iOS, they're free to do so without have to re-encode the files!
\nCompanion Ads and Banners
\nThe same can't be said for companion ads, however. Companion ads are static or animated images that you see outside the video player area (usually in another div on the page). The problem is that these ads, in my experience, aren't static images and usually animated Flash. This means that advertisers and content producers need to work together to create new companion creatives and serve them up appropriately for Flash or iOS.
\n", "date_published": "2011-09-09T00:00:00Z" } , { "id": "https://benfarrell.com/blog/2011-09-09-html5-vs-flash-video-choose-wisely-part-7-whats-happening-on-the-server-side/", "url": "https://benfarrell.com/blog/2011-09-09-html5-vs-flash-video-choose-wisely-part-7-whats-happening-on-the-server-side/", "title": "HTML5 vs Flash Video: Choose Wisely Part 7 – What's Happening on the Server Side", "content_html": "So we talked about how Apple supports both progressive and streaming video with their HTML5 video tag. We also briefly discussed how Flash Media Server serves up its OWN streams. What about Apple's HTTP Live Streaming? What kind of magic is that?
\nTurns out that its quite simple. Take a video file, after encoding it for HLS you'll end up with a directory of files. First off there is a table of contents file. This table of contents files points to MORE table of contents files, and these point to video file segments.
\nThe video segments are most likely h264 encoded segments. Each one has a "ts" file extension (myfile.ts). "TS" stands for transport segment. Each segment will most likely be small, about a megabyte or so and contain several seconds worth of video (this is entirely left up to the encoder settings).
\nThe table of contents files sound a little hacky, but they work well. Apple has gone ahead and adopted m3u8 playlist files for streaming video. What's that you ask? What's m3u8? Well, if you crack open Winamp or iTunes, you can create a music playlist. Both m3u and pls files are pretty standard for holding playlists of songs. The m3u8 spec is an enhanced version of this (and still used for music playlists!). It makes sense though, you're really just assembling a playlist of video segments.
\nI also mentioned playlists in playlists didn't I? Well, the master m3u8 file will typically hold several different bitrate references (for if a user wants high definition, low definition, or anything in between). This file will hold references to other m3u8 files and list what their bitrates are in bytes per second.
\nThe reference m3u8 files will be the ones that actually list the segments, examples are listed below.
\n\nFirst is an m3u8 file referencing OTHER m3u8 files for the different bitrates:
\n#EXTM3U #EXT-X-STREAM-INF:PROGRAM-ID=1, BANDWIDTH=200000 http://ALPHA.mycompany.com/lo/prog_index.m3u8 #EXT-X-STREAM-INF:PROGRAM-ID=1, BANDWIDTH=200000 http://BETA.mycompany.com/lo/prog_index.m3u8 #EXT-X-STREAM-INF:PROGRAM-ID=1, BANDWIDTH=500000 http://ALPHA.mycompany.com/md/prog_index.m3u8 #EXT-X-STREAM-INF:PROGRAM-ID=1, BANDWIDTH=500000
\nNext up is an example of the actual m3u8 file that holds all of our file references. #EXTM3U #EXT-X-TARGETDURATION:10 #EXT-X-MEDIA-SEQUENCE:0 #EXTINF:10, no desc fileSequence0.ts #EXTINF:10, no desc fileSequence1.ts #EXTINF:10, no desc fileSequence2.ts #EXTINF:10, no desc fileSequence3.ts #EXTINF:10, no desc fileSequence4.ts #EXTINF:10, no desc ..... #EXT-X-ENDLIST
\nLive Playback Limits on HLS
\nIn the example above, notice the line #EXT-X-ENDLIST. This command marks the end of the file. This an important piece of info! It tells Quicktime that there will be no more segments after this, and what we have in the file is the entire stream.
\nWhat if this line didn't exist? Well, it turns out that the absence of this line indicates to Quicktime that the m3u8 file is a live stream. In this regard, the file has no end. With a VOD (video on demand) stream, Quicktime only needs to query this table of contents once, and the player is secure in it's knowledge that there is nothing else to load. For a live stream, the m3u8 file is repeatedly queried to find out what the latest segments are so the player can keep up with the most recent segments.
\nThis is a nifty little system - but there's a problem. Typically, a live event will allow the end user to watch in a 30 minute or so sliding window. The user can't seek backwards more than 10 minutes or so. These older segments are dropped from the m3u8 file such that the m3u8 file only contains a handful of segments.
\nThe problem is if you need a long stream. My last project had a need to run clips from a large 14 hour stream. When playing through HTML5 on a web page, the stream broke down after it grew to 40 minutes or so. The problem was that the growing number of entries referencing transport segments made the file grow exceptionally large after a while. If I recall correctly, a 10 hour stream's m3u8 file was around 600kb for me. Multiply that by 3 for the other bitrates I had. So eventually, Quicktime was constantly querying over a megabyte of playlist info - and it had trouble keeping up with playback!
\nI came up with the solution of having our encoding team put a fake end on the m3u8 file even though it was live. This caused the file to only be loaded once even though the live stream was continually playing. It doesn't work for a lot of situations since you're not able to play the most recent stuff since you loaded the file, but it worked for our specific needs with our site!
\nMiraculously, it seems that this is only a problem in HTML5, and not a problem if you're just running Quicktime natively or in an app.
\nOther Streaming Services
\nI don't have any experience with Silverlight, but IIS's Smooth Streaming functions pretty much the same way. Instead of an m3u8 file, Smooth Streaming has an XML format for it's table of contents.
\nWe discussed how Flash Media Server worked in a previous post in this series - but to recap, it can generate these segments on the fly and serve them over RTMP, HTTP and for Flash or iOS.
\nPlayback in Flash doesn't have the same length issues that iOS HTML5 has for live streams. It plays things back fairly well! HLS and SmoothStreaming though, have a pretty important feature on Flash - bitrate switching happens automatically between the server and the player. You don't need to program anything extra in your code to make bitrate switching work - and it does work very smoothly! You'd be hard pressed to notice the transition.
\nFlash recently implemented a smooth stream switchover as well. You as a developer, must measure bandwidth and make the switch yourself (though OSMF will do this automatically for you). Once you call the "play2" command, it will START making the switch, but it will wait until the next video keyframe to make the switch so its as seamless as HLS and Smooth Streaming make it.
\n", "date_published": "2011-09-09T00:00:00Z" } , { "id": "https://benfarrell.com/blog/2011-09-09-html5-vs-flash-video-choose-wisely-part-6-apples-html5-vs-normal-html5/", "url": "https://benfarrell.com/blog/2011-09-09-html5-vs-flash-video-choose-wisely-part-6-apples-html5-vs-normal-html5/", "title": "HTML5 vs Flash Video: Choose Wisely Part 6 – Apple's HTML5 vs Normal HTML5", "content_html": "If you are interested in pursuing HTML5, I imagine the reason why you are doing so is to support iOS. As we all know, iOS doesn't support Flash, so to play video we'll need to take advantage of HTML5 video.
\nHTML5 is fairly normal via Safari on OSX. Safari is just another browser, however the video tag is handled by Quicktime. It looks like we're trading one plugin for the other here!
\nThere's probably nothing wrong with using Quicktime here in OSX. If Apple chooses to use one of their oldest video technologies to playback video in their browser on their OS, no big deal! It really probably a matter of semantics anyway - could you consider the h264 decoder in Chrome a plugin? Yes and no - its a fine line.
\nOn Windows, however, if you use Safari, you'll need the Quicktime plugin installed! So, here we're REALLY talking about a plugin. In this situation its both a plugin AND HTML5 at the same time.
\nAs it is Quicktime, we can use the video tag to play whatever Apple can normally play with Quicktime. In fact, we can play Apple's HTTP Live Streaming (HLS). I think this leads to a great deal of confusion and is a pretty big bait and switch on Apple's part.
\nFirst they tell us "who needs Flash?! Its slow, buggy, and HTML5 can replace it!". Then they live stream their developer event in HTML5, but surprise! Only Apple's brand of HTML5 can stream it, because it's really Quicktime under the hood. Even more than that, streaming video isn't even part of the HTML5 spec! So, it should be quite awhile before we'll see streaming content come to browsers natively.
\nSome Apple playback quirkiness
\nI really haven't seen much in terms of weirdness on Safari OSX. I haven't had to play with it much, but the video tag works pretty much as advertised here (since it's webkit, it's just like using Chrome in fact).
\nBut that's not really what matters - on OSX you have the option of Flash too. On iOS, you either need to go HTML5, or write an app. There's no other option!
\nSo, how does the video tag fare? There are some major things here to be aware of. Apple has decided that we can't be trusted with the autoplay feature of the video tag, so they have disabled it. This means that you cannot automatically play a video after a page loads. There needs to be some sort of user intervention here to kick off the video. This will usually involve a play button overlaid on top of the video that the user clicks to start playback. It's a little heavy handed in my opinion, but the reasoning behind this is to not let malicious people use your data for things that you don't want - like ads.
\nUnfortunately, this prevents some major barriers for advanced video applications. Something as simple as playing midroll ads (ads that appear in the middle of content at a specified ad break) is made complicated.
\nMidroll Ads
\nSee, one problem with midroll ads is that you must stop your content, flip over to the ad, and then after the ad, resume content again. I've done this in Flash and am currently doing this in iOS/Objective-C. In my experience, the best way to do this is to use two separate video objects. With your main content in one video object, when an ad comes on, you can pause this and spawn a new video object overlaying it on top of your main content. Play your ad in the new video object, and then when you're done, kill the object and resume your main video.
\nWith this game plan, you don't need to suffer through a rebuffer of your main video content after your ad. If using the same video object, you'd need to not only load the video again, but automatically seek to the spot right after the ad break.
\nUsing two video objects like this isn't really feasible. For each video tag you use, you'd need to tell the user to manually click play on each one. When playing an ad - you're adding insult to injury like this!
\nFurthermore, another quirk I've seen is when you hide a video tag on iOS using CSS and either hidden or display:none. This consistently crashed my iPad browser when I attempted this. So maintaining 2 video tags, but hiding one isn't an option!
\nIn essence, midroll ads seem to be stretching the capabilities of HTML5 on iOS!
\nAutomatically Seeking
\nYet another quirk I've seen is attempting to automatically seek into a stream a certain number of seconds (which you'd need to do when returning from an ad, or just playing a clip of a larger video). The problem is that when you initially load a video resource, it can take some time before you can interact with it and tell it to seek.
\nThe initial run up to playback includes the video object becoming ready, the asset being loaded enough to start playback, and finally as more of the asset is loaded, and finally it becomes seekable. The problem is that this process can take a second or so. In the MEANTIME you can't interact with the video and tell it to seek. You just have to sit there and watch it playback a second or half second of video. What should be an automatic seek becomes an automatic seek with a small video glitch at the beginning.
\nPlayer Chrome
\nYour customers like customization don't they? Some would like some nifty custom graphics to show up wouldn't they? A custom lime-green playback button to match their brand maybe? It's completely possible on the desktop with HTML5 - simply hide the controls and create your own, positioning them over the video with CSS. The problem is that this is a bit problematic in iOS. See - in my experience, you can't overlay things on top of a playing video. Well actually you can - but the video window will suck in all your mouseclicks. This means that while you can overlay your own controls on iOS, they probably won't function. Nothing can obscure the video and be perfectly functional!
\nFurthermore, you always get the same UI when you run either on an iPhone or an iPad in fullscreen mode. You get Quicktime's native controls - your page is gone (in the background) and you are left with nothing but the video playing in the Quicktime application. Fortunately enough, Javascript can still operate in the background and control your video.
\nVideo API Events
\nI've also noticed that events can be a little unreliable on iOS, like listening for "onCanPlay" or when the player is ready. I've had to setup a timer that fires every half second or so to check for readiness and playback state in my own project to accommodate this problem.
\n", "date_published": "2011-09-09T00:00:00Z" } , { "id": "https://benfarrell.com/blog/2011-09-09-html5-vs-flash-video-choose-wisely-part-10-other-options/", "url": "https://benfarrell.com/blog/2011-09-09-html5-vs-flash-video-choose-wisely-part-10-other-options/", "title": "HTML5 vs Flash Video: Choose Wisely Part 10 – Other Options", "content_html": "Lets face it, Apple really stuck us in a bad situation. While HTML5 is really slick on the desktop (in supported browser with supported file types), its pretty clunky on iOS. And Flash is ALSO really slick on the desktop, but doesn't work at all on iOS.
\nSo what we are left with are 2 things that work really well on the desktop, but no common thing that works well on Android, iOS, and the desktop.
\nWhat can we do?
\nNative iOS
\nSimply the best option on iOS now is to sign up for a developer's license and make an app. iOS offers two different video player frameworks MPMoviePlayer and AVPlayer. MPMoviePlayer is the simplest to get started with, and offers you playback through the Quicktime UI. AVPlayer is harder to get started with and you are forced to create your own UI, but you have more control over video playback.
\nAIR for Mobile
\nWhat may be the best solution is to build a Flash player with HLS streams on iOS and normal HTTP or RTMP streams on Android or the desktop. While it's true that Flash isn't supported on iOS, Flash Builder 4.5 can compile your existing project out to iOS and Android applications. All Flash video is supported in AIR, Mobile AIR and the web Flash player - except for iOS. So you'll need the two different streams.
\nCorona and other 3rd Party Packagers
\nCorona seems to be a very popular way to create cross platform mobile apps these days. There are other packagers as well like Titanium, Appcelerator and more. Last I looked they don't handle video, or don't handle video very well. Probably the best is Corona which does allow video to play, but it needs to launch the video in a modal window or fullscreen, and you have little control over it. As cool as these solutions are, I don't believe that they handle video all that well.
\n", "date_published": "2011-09-09T00:00:00Z" } , { "id": "https://benfarrell.com/blog/2011-09-09-bolt-talks-logo/", "url": "https://benfarrell.com/blog/2011-09-09-bolt-talks-logo/", "title": "Bolt Talks Logo", "content_html": "Tim Cunningham, of www.cfmumbojumbo.com, needed a logo for his podcast/radio show where he does quick interviews featuring one guest per show.
\nI helped him out, starting with a few variations on a concept I came up with.
\n![\"bolttalks\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2011/09/bolttalks.jpg\")
Tim picked a concept after asking around, and I finally added some color and depth while coming up with a wide version and a square version:
\n![\"bolttalks_color\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2011/09/bolttalks_color.jpg\")
We've discussed in part 4 of this series that Flash can get a little complicated. It's true, but it mostly boils down to the different video formats you can play. Let's talk about those formats.
\nProgressive Video
\nIf you are clueless about what the difference is between streaming and progressive video - chances are the video you know and love is said to be "progressive". What does this mean?
\nWell, if you hit a single video file on a server somewhere and start downloading it, you can actually watch the video before it finishes downloading. You are using it "progressively" in that you are watching it while download is in progress. This happens in much the same way as if you had an image loading on a slow connection and you see it start at the top and fill in as it gets loaded.
\nOne of the problems I've seen from progressive video files (at least when playing from Flash) is when the MOOV Atom is placed at the end of the file by the video encoder. Think of the MOOV Atom as a block of data in the file that holds info about your video...like duration, bitrate, frame rate, encoding specs, etc. If this is place at the end of the file, the player doesn't know the details of your video file, and won't play it UNTIL it gets this block of data. Since it's at the end of the file, users need to wait until the entire video file is downloaded so that the video player knows exactly what it's playing.
\nStreaming Video
\nSo, speaking progressively, video is one large video file that you download and cache to your computer (or at least that's whats happening behind the scenes in your browser) for playback.
\nOn the other side of the coin, we have streaming playback. Typically a streaming file will be served from some sort of media playback server as a lightweight container file that serves as a table of contents for many smaller video segment files. The lightweight container file will be loaded once up front (or continually if it is often updated like when you are connected to a live stream). Your player will then be smart enough to know what segment files are needed to playback video. It will go out and grab these segments, and then throw them away when you're done. A player will keep a "buffer" that may or may not be able to be set by the player of how many seconds of video to keep (and therefore how many video segments). If you have video buffered, you can easily seek back and forth to these points instantaneously. If on the other hand, you seek outside the buffer, the player will have to read the table of contents container and lookup which video segments to fetch and buffer for you.
\nThe Flash Media Server will hide all the complexities from you, though. Usually, you'll just connect to a path that starts with rtmp:// and ends with myfile.mp4. For all YOU know it's just a file, not chunked up into tiny segments. And actually it sort of IS one big file. Flash Media Server will just tranform this file into a stream for you on demand, so its easy to manage and still works like a stream.
\nFlash Media Server, in recent days, has actually started supporting streams over HTTP for Flash and iOS. This means that if your company doesn't want to deal with serving stuff over RTMP, you can just use HTTP streams and Flash 10.1 (and OSMF if you don't want to program support yourself). You can use your same server and files to send a stream to iOS! Kind of a timesaver if you need to support both, instead of managing all 2 separate servers/encoders.
\nUse Cases for Each
\nWhy use streaming vs progressive? Well the major reason is what we in the business call "long-form video". Long form video is typically 20 minutes to a few hours long, while "short-form video" is under a minute or a few minutes long. YouTube clips are perfect examples of short form video. And they are also decent examples of why the overhead of having a streaming service and player is completely unnecessary. If it's only a few megabytes, then it really doesn't take long to load and cache the whole video does it? And in the end, when you close your browser you'll only have a few MB from it in your cache or in your memory.
\nLong-form, on the other hand, can be several hundred megabytes or even a couple of gigabytes depending how long the video is and what quality it was encoded at. Not only will you be left with a behemoth of a file in your RAM or cache, but it you can't seek forward to skip over pieces until that much has been downloaded.
\nAnother benefit of streaming is the ability to switch between bitrates or cameras. If your video player thought that it was having difficulty keeping up with your HD file, ideally, you'll switch to a lower resolution video. It's REALLY difficult and pointless to do this on a progressive file. You'd have to latch on to a different file, and then seek to the position you left off at....which means you'd have to progressively download the file up to the point where you left off which could take several minutes! With a stream, all you have to do is tell the server to start sending you different file segments. It will do so, and the player will never be the wiser, cause all it sees is new segments coming in - it doesn't really matter if they are from a different source file.
\nSo that coupled with a server's ability to offer DRM is a huge reason why TV shows and movies are all streaming. While there are many DRM solutions available, one fact is that when you segment all your files like this (even without DRM), its a pain in the butt to download and assemble them all if you were so inclined to steal content. Flash Media Server obfuscates the stream even more by serving it over RTMP. Since it's an entirely different protocol than HTTP, most standard web dev tools won't even see the files come through and its even more difficult to steal!
\nContrast this with YouTube. Most of the time, I can go in and open up my Chrome developer tools, watch a big file come in, and copy the source URL. I can then download the file and rename to a mp4 or flv file and then play it on my desktop. BAM! I just stole content and it took 30 seconds.
\nSomething in between (pseudo-streaming)
\nYou may have noticed recently in YouTube, where it shows you a red line in your progress bar to indicate your downloaded buffer. In years past, you couldn't seek beyond this. Well, now you can! How is this possible?
\nIt's what we call "pseudo-streaming". Basically it acts like a progressive download, but with trickery it can have the main benefit of streaming which is to seek anywhere you want in the file regardless of how much downloaded.
\nIt's do-able because many servers (even if just a simple Apache server running PHP) allow the browser request to specify a byte range. Typically when you make a file request from your browser you ask for the file, and it ASSUMES you want all the bytes of the file starting at the beginning of the file. What pseudo streaming does is to ask for the file, but ask for the bytes at a certain position in the file and not to start at the beginning.
\nWhat's Possible in HTML5 vs Flash
\nPseudo Streaming is actually very popular these days because the HTML5 video tag supports it natively. To use it, you as a developer don't have to do a thing beyond allowing your user to seek (as well as having a server that supports byte ranges). Contrast this with Flash - where you'd need to jump through hoops to get this working. Flash doesn't support byte ranges with video or audio, which means you'd have to load the bytes MANUALLY, and then append the bytes onto the active video stream. Not only does this suck to code, but appending bytes to a stream wasn't even supported until Flash Player 10.1.
\nOf course, progressive is supported in both HTML5 and Flash. Flash supports a few different encoding types including mp4/h264 just like HTML5 normally does (depending on the browser).
\nHere's the bombshell though...HTML5 does not support actual streaming, and there is no official proposed spec for it! I bet you don't believe me because of a certain large, fruity corporation. It's true though! More on that in the next part. But the long story short here, is that I can't imagine recommending HTML5 for long form content, doing 2 hour movies with a progressive download would be beastly (note that there may be clever ways, I just don't know what they would be beyond artificially breaking up the video files and leaving gaping holes in playback while loading the next segment).
\n", "date_published": "2011-09-08T00:00:00Z" } , { "id": "https://benfarrell.com/blog/2011-09-08-html5-vs-flash-video-choose-wisely-part-4-how-to-basic-flash-playback/", "url": "https://benfarrell.com/blog/2011-09-08-html5-vs-flash-video-choose-wisely-part-4-how-to-basic-flash-playback/", "title": "HTML5 vs Flash Video: Choose Wisely Part 4 – How to: Basic Flash Playback", "content_html": "The interesting thing about Flash is that before some recent projects, it was much harder to get a basic player going. Video playback is composed of three basic elements: the NetStream, the NetConnection, and the Video object.
\nTo start playback, you'd need to create a NetConnection, connect to a server if streaming or null if a normal progressive file, start a new NetStream with the NetConnection, and finally attach a the NetStream to a video object.
\nconnection = new NetConnection();\nconnection.connect(null);\nstream = new NetStream(connection);\nstream.client = this;\nvideo = new Video();\nvideo.attachNetStream(stream);\nstream.play("myfile.mp4");\naddChild(video);\nNot only that, but you'll need to add the video to the stage, and manage the aspect ratio of the video properly. You'll also need to publish a SWF, and embed it on the page.
\nIt's a heck of a lot more complicated than a simple HTML video tag, isn't it?
\n\nAdobe's Open Source Media Framework
\nA couple of years ago, Adobe set out to make things less complicated - to create a framework that can simplify video playback as well as provide an easy plugin architecture for ads, reporting, and more. This framework can handle any video type that Flash can handle without you needing to worry about what type of video you're playing.
\nLet's check out how to get a simple video up and running:
\nplayerSprite = new MediaPlayerSprite();\nplayerSprite.media = new VideoElement( new URLResource( "myvideo.mp4" ) );\naddChild( playerSprite );\nYou'll still need to compile and embed on a page as well. This is only 3 lines of code as opposed to the 8 without OSMF. Its not necessarily the amount of lines of code I'm concerned with, it's that to really know what's going on, you need to know what the NetStream, NetConnection, and Video objects all do. It can take a bit of research to put all those pieces together! In other words, OSMF is more intuitive to me - you create a playback sprite, and then assign some media to it
\nThe extra good thing is that we can handle most file types. We'll get back to this in a bit, but as you use streaming media, worry about hooking up a control bar, things get exponentially more difficult. OSMF makes this nice and tidy for you. I must warn you though, as you start doing weird things with your video playback, you'll need to learn the language and architecture of this framework. Normally though, things are quite simple to use through the framework's API.
\nStrobe and Flash Media Playback
\nCan it get easier maybe? Answer is yes! All levels of ease seem to be taken care of - from the insanely detailed and comprehensive to having a ready made SWF pre-compiled for you and ready to use!
\nThe next level here is the Strobe Media Playback project. This project is already compiled and ready to go...ready to be embedded on a page. It comes with some external configuration and external skinning options if you need to change anything around. In addition to the compiled SWF, it comes with all the source code to the player as well - so if there's anything you need to change with code, you're welcome to make the change and compile again!
\nCan it get easier? Yes! There's Flash Media Playback. If you follow the link, Adobe brands it as "Instant Video Playback on your Website". Flash Media Playback is an extension of Strobe Media Playback - however, you don't download any files. Adobe hosts the player, you simply grab the embed code and perhaps make your own configuration file that you point it too to handle the customization aspects you need. Adobe offers a configuration and test playback page for you to get started.
\nOther 3rd Party Players
\nThere are plenty of players out there in the same vein as Flash Media Playback, and most have probably been around longer! The most popular ones that I'm aware of are JWPlayer, FlowPlayer, and the BrightCove (which has an entire video publishing platform)!
\nTypically you'll embed these on your page and pass in one or several "flashvars" to tell it what video to play. I don't have much experience with these 3rd party players, but most cost money if you'd like to use them commercially.
\n\nAs you can see, using video in Flash can range from drop dead simple to mind boggingly complex. More on why this is in the next part of this series. What's good is that there are several projects you can take advantage of to get simple playback on your website with no or minimal Flash experience....without even having to buy or touch a compiler even!
\n", "date_published": "2011-09-08T00:00:00Z" } , { "id": "https://benfarrell.com/blog/2011-09-07-html5-vs-flash-video-choose-wisely-part-3-extra-help-with-html5/", "url": "https://benfarrell.com/blog/2011-09-07-html5-vs-flash-video-choose-wisely-part-3-extra-help-with-html5/", "title": "HTML5 vs Flash Video: Choose Wisely Part 3 - Extra Help with HTML5", "content_html": "In part two we discussed a good deal of how to use the video tag and how (if we're ambitious) we can code up our own controls. We also discussed the fact that the HTML5 video tag might not be an option if you're using an older browser. And maybe Firefox/OGG video isn't part of your workflow. In either case, you need to opt out of HTML5 and opt into Flash (or vice versa - whatever floats your boat).
\nLuckily there are a few projects to help out with that!
\nModernizr
\nIf simple feature checking is your game, Modernizr is a pretty big help. Simply use Modernizr.video in your Javascript to detect if you can play video. Modernizr.video.ogg, Modernizr.video.webm, Modernizr.video.h264 will do well to test if you can support the various formats.
\nNot Modernizr
\nIf you'd rather not use Modernizr, you can easily instantiate a new, empty video tag in your document (attached to nothing). If the video tag object you've created support the "canPlayType" method, then you have a legitimate video tag!
\nFor example:
\nvideo = document.createElement('video');\nvideo.canPlayType('video/mp4');\n\nNo Javascript
\nYou can also take the no Javascript route. Kroc Camen setup a nice bit of HTML that does "detection" in a Javascriptless way. And then Jonathan Neal took it one step further and made a nice little "Video For Everybody Generator" application based on the "Video For Everybody" idea.
\nhttp://sandbox.thewikies.com/vfe-generator/
\nThe basic idea is that you''ll hit the video source tags first, and if successful will start video playback on the given source. If you make it to the Flash object/embed tag, you've failed all HTML5 tests and you get Flash playback!
\nVideo Frameworks
\nThe simple stuff leads into the heavyweights. The Javascript frameworks. The "Adobe Video Widget" is as good a place as any to start. You can grab it from the Adobe Widget Browser and add it to Dreamweaver....drag and drop style!
\nThis widget is indicative of quite a number of frameworks out there. Depending on your preference, it will prioritize HTML5 and fall back to Flash. Even though it's branded Adobe, the Javascript player and Flash player are both created by a company called Kaltura.
\nThe interesting thing about a lot of these frameworks, including Kaltura, is that when you peruse the source code it seems obvious that they intend you to use these frameworks as sort of a slapdash HTML5 player which falls back to Flash. In my opinion, many of these frameworks aren't friendly to you digging around and extending things in major ways.
\nI had previously scouted a few different player libraries for a project - here's my rundown of findings:
\nGiven all these options - VideoJS seemed the very best for my purposes. I didn't want to fallback to Flash since I was doing an iPad site and this was easy to turn off. I had a fair number of custom features I wanted to implement as well, and it was very easy to tap into it's underlying framework and add new features like advertising and reporting.
\nWhat I really dug was the easy of skinning. It comes with a base video skin along with 3 skins which extend the base skin. It was quite easy to skin how I wanted!
\nAll in all thought, you probably won't need to go into the level of customization that I needed, so whatever works easiest for your purposes is best - but it seemed to me that VideoJS was helpful right out of the box!
\n", "date_published": "2011-09-07T00:00:00Z" } , { "id": "https://benfarrell.com/blog/2011-09-07-html5-vs-flash-video-choose-wisely-part-2-playing-with-the-video-tag/", "url": "https://benfarrell.com/blog/2011-09-07-html5-vs-flash-video-choose-wisely-part-2-playing-with-the-video-tag/", "title": "HTML5 vs Flash Video: Choose Wisely Part 2 - Playing with the Video Tag", "content_html": "Simple playback
\nSimple playback is just that! Simple. This can be a very good thing. Offering a video up via the tag allows for easy playback. Easy for the user, and easy for the page developer!
\n<video width="640" height="360" controls src="myvideo.mp4"></video>\nSee that?! It's just like an image tag! Put a source, height, and width on that tag, and you've got video!
\n\nNative vs Custom Controls
\nI avoided mentioning the part of that tag that reads "controls" when I was laying out the video tag above. Basically the presence of this tag means that you're telling the browser: "Yes I would like you to displayo your standard video player controls". Many times developers will write controls="controls", controls="true", etc. All mean you want the browser's native video controls on your video.
\nNormally, these controls include play and pause buttons - as well as a volume slider and progress/seek slider. These controls will look different as you bounce between Chrome, Safari, Firefox, IE, and devices.
\nSo, if a consistent look/feel across devices and browsers is in your customers best interest, its time to ditch the native controls. Likewise, if your customer has additional buttons that they would like to add to the control bar, such as an "information about this video" button, it might be time to ditch the native controls.
\nBasically, once ANY customization is required to the players control bar, you've crossed into building everything yourself.
\nWhat's involved in doing custom controls? Well, now we're talking about creating individual buttons and sliders ourselves. These buttons and sliders need to hook into the video tag's Javascript API. After creating our controls, we'll need to stylize them with CSS to overlay them over the video and act like a real control bar.
\nThis isn't necessarily a bad thing, the video tag has a pretty decent API. It's a little bit of work, but you didn't think you were getting a custom skin for free right?
\nJavascript API
\nCSS is beyond the scope of this post - but needless to say, you'll most likely be relatively or absolutely positioning your control bar and making sure the z-order is higher than that of the video. You'll also probably use some nice jQuery animation for when the user mouses over the video area to auto-show the control bar.
\nOnce you've fleshed that out, you'll be wiring up your buttons!
\nObviously your play button would call video.play(); Or would it? What if the video was already playing - it might call video.pause() instead! Play/pause buttons are usually grouped together in a playback toggle button. You'll need to handle this logic yourself as well as update any buttons to show the appropriate icon.
\nWhether the video is playing or paused can be checked via the video.paused property. You can check this boolean to determine the status.
\nHandling your volume control should be pretty easy if you just wanted to use a slider. The value of the slider could effect the video.volume property. The volume could be anything between 0.0 and 1.0.
\nNext up is the current time of the video and the ability to seek/scrub anywhere in the video. Again, you could start with a simple slider. You can get the duration by asking for the video.duration property, and the current playback time of the video by asking for the video.currentTime.
\nThe video.currentTime is also a settable property. Setting this to a specific number (in seconds) will cause the video to seek to that time.
\nBe careful though! If your video is just beginning to load, it may not be seekable. You can find out what parts of your video are seekable by using the video.seekable property. You can test this by using the "TimeRange" object that the seekable property gives you back when you ask for it. Testing if your desired seek time falls before video.seekable.end is a good indicator that your seek will work as advertised. I actually like to test if the seekable object returns an object with a length over zero just to make sure it's not empty and not even initialized enough to tell me if it's seekable.
\nThe video.buffered property will also return a "TimeRange" object and if you're doing anything advanced, knowing what video is buffered could be very useful info!
\nNext up we have events! You may have thought that you can get the video.currentTime to find the time of the video and update the playback slider to the correct position. But should you include a Javascript timer to poll the current time to see if it changes? That's not necessary because you can add an event listener on the video object to listen for "ontimeupdate".
\nThere are several other events you can listen for that will help you out...but first, I'd like to save you an hour of banging your head on the desk!
\nWhen placing the event inside the tag itself, you use "ontimeupdate", like so:
\n<video width="640" height="360" controls src="myvideo.mp4" ontimeupdate="myFunction()"></video>\nHowever, when adding an event listener, the event type doesn't contain the "on", like so:
\nvideo.addEventListener("timeupdate", funtion() { my script } };\nLike I said, there are a few other events you can listen for. The event type of "onplaying" can alert other parts of your application when the video starts to play. The event type of "oncanplay" will tell you when the video is ready to start playback. The event type of "onloadeddata" is another good one - it indicates that the data for the current frame is loaded, just not necessarily that we can continue playback. Another good event type is "onerror". You can listen to this error when the video can't be played, and show something nice to the user to redirect them to something else.
\nSome shortcomings
\nAs far as simple playback goes, the video tag is simple...and it works. It's actually pretty decent and comprehensive for many situations.
\nAre there any problems? Well, yes.
\nUsing the tag as defined by the W3C is easy, but it's not reliable! The W3C spec defines using a video file for playback - it talks about how you can interact with the video and what events you can listen to from the video.
\nProblem is, the spec doesn't actually indicate what TYPE of video the tag can play. Browser makers are left to fend for themselves when it comes to implementing video playback for any given video format.
\nFlash popularized the h264 specification. Many argue its the most compact and efficient. Problem is that it's patented by MPEG-LA, a group of companies who all hold various patents on the standard. The group is comprised of Apple, Microsoft, Panasonic, Sony, and 22 other patent holders.
\nTo use h264 encoding or decoding, users must pay a license fee to the MPEG-LA group. Since the Flash plugin has h264 decoding (and now encoding) built in, it's Adobe's responsibility to pay royalties on the specification. Many speculate that the code to implement h264 encoding is the reason why Adobe can't release the whole flash player source code publicly.
\nAnyway, the point is, that each browser must pay up if they want to use h264. Internet Explorer, Chrome, and Safari do. Chrome was threatening to pull the plug on it any day now in favor of their VP8 codec that they open sourced such that anyone can use it. Ogg is ALSO free and open source, which is why it's the only format that Firefox supports.
\nWith Google backing VP8, and Adobe implementing it in Flash, it seems the most promising unless everyone just decides to continue paying MPEG-LA.
\nSo now we have a problem - how do we tell our video tag to play multiple formats based on the browser/device? Luckily, the video tag supports multiple source tags:
\n<video width="640" height="360" controls >\n <source src='video.mp4' type='video/mp4'>\n <source src='video.ogv' type='video/ogg'>\n</video>\nThis solves the playback problem, but we still have to encode our videos twice over! Or 3 times over if Google ever decides to drop h264 and go with VP8...then we're encoding h264, VP8, and ogg.
\nAdditionally, we're stuck using only those browsers that support the video tag in the first place. This issue is becoming less of a problem with IE being the real drag since only IE9 supports it.
\nLet's talk about the reality of this sticky situation in the next post!
\n", "date_published": "2011-09-07T00:00:00Z" } , { "id": "https://benfarrell.com/blog/2011-09-05-html5-vs-flash-video-choose-wisely-part-1-war-of-the-buzzwords/", "url": "https://benfarrell.com/blog/2011-09-05-html5-vs-flash-video-choose-wisely-part-1-war-of-the-buzzwords/", "title": "HTML5 vs Flash Video: Choose Wisely Part 1 - War of the Buzzwords", "content_html": "A Little History of Web Video
\nLet's go down memory lane a little bit. RealPlayer, Quicktime, and Windows Media Player were the big web video players. The problem with these options, though, was that you didn't really own the video as a site creator. All of them seemed to impose their own UI on people, and video was in a box that you couldn't do well to brand or easily bring to the web.
\nEach technology didn't dominate the marketplace, so it was hard for someone that wanted to create video to know whether to create their content in wmv, mov, or....whatever realplayer was.
\nBack around 2000 or so, I personally saw RealPlayer as a trainwreck...always crashing, bad at buffering, installing extra spyware, etc. Windows Media was decent, but since I was a cross platform CD-ROM developer at the time, Quicktime was the best to author in since there were players on Windows and Mac.
\nIn March 2002, Macromedia's Flash Player 6 came out. Suddenly web video was a reality in a big way - and not only that, but it didn't require any plugins beyond Flash itself. Folks could even put video in a project and do whatever they wanted with it. They could make their own controls, and use video itself in an interactive way. We weren't limited to playing our video through someone else's player.
\nFlash Player version 5 at the time had 98% penetration numbers, so Macromedia was bringing video to a plugin that virtually everyone already had (they just needed to upgrade). So, virtually overnight, us video creators had a ubiquitous platform that anyone could use.
\nDirector had given us something similar earlier in the form of the Shockwave plugin, however that plugin never really had the install base of Flash, especially given that it was widely used for CD-ROM development and wasn't really able to shake it's place in life and get traction on the web. The other problem with Shockwave was that it still required a plugin (plugin to the plugin) to play video. We had options to play video in whatever format suited us, and in my experience, Quicktime was the most popular. Video wasn't as moldable as Flash though. If I remember correctly, it was still standard to see the controls on the player, and we weren't as free to do what we wanted. Additionally, the video plugin didn't layer the video like Flash did. It really just created a graphical hole in your browser, and the video would play in this rectangular hole, and things could not be overlaid on top of it.
\nYay Flash Video!
\nIn just 3 short years after Flash Player 6 was launched, a little company called YouTube was formed. YouTube, though, is just a good example of what Flash 6 did to pave the way for a revolution of online video. It enabled technology to get out of the way, and get brand owners to do what they wanted with their own video!
\nMacromedia rode the popularity surge and created more products around the Flash platform including media encoders and media servers. Folks could now do actual streaming video with Flash Media Server released shortly after Flash Player 6.
\nAn ability to stream on demand and live video led to an even bigger explosion of live events and long-form video content. With several DRM offerings on top of this, folks could start to see popular movies and tv shows streamed on the web.
\nDespite any objections to Flash recently, I do think that it's hard to argue that Flash accelerated public consumption of online video in a big way.
\nEnter the Video Tag
\nYou may have heard of several "Flash Killers" come along since 2002. In this, I mean the ability to deliver rich media on the web (including video). Microsoft's Silverlight released in 2007 was probably the most formidable competitor. It's "Smooth Streaming" server side component was most likely the reason why the 2008 NBC coverage of the Olympic games used Silverlight, and why Netflix uses Silverlight today.
\nThe reason Smooth Streaming was so awesome was for a few reasons. First, streams would upgrade and downgrade depending on the user's bandwidth all on the server side, and was a completely smooth experience (not requiring client side logic to handle this). Secondly (and this is big), streaming was done over HTTP, something that large organizations are better equipped to deal with over Flash Media Server's RTMP protocol.
\nDespite the fact that Netflix still uses Silverlight, the buzz has significantly died down - not least of which due to Microsoft's handling of it!
\nThe buzz picked right back up in April 2010 with Steve Jobs' infamous "Thoughts about Flash" memo. It launched a sort of web "arms race" gearing up for the still unfinished specification that was HTML5.
\nIn the unfinished specification, was the notion of the video tag. On the surface, a video tag sounds simple - just like putting an "img" tag in the browser to display an image. Underneath though, the video tag is more complicated as folks fight out the underlying video codecs.
\nIn 2010, we weren't NEARLY ready for the video tag. A year later, we still have the same problems, but given recent browser support and all the hype, we've been rocketed into a position where the video tag is a viable solution in many ways - but still overhyped enough that people think it's as robust as the Flash ecosystem and ends up failing when people try to use it in these advanced ways.
\n", "date_published": "2011-09-05T00:00:00Z" } , { "id": "https://benfarrell.com/blog/2011-09-04-live-instrumentation-in-flash-part-7-beyond-flash/", "url": "https://benfarrell.com/blog/2011-09-04-live-instrumentation-in-flash-part-7-beyond-flash/", "title": "Live Instrumentation in Flash Part 7 - Beyond Flash", "content_html": "At the heart of all our audio generation lies one simple Flash feature - the sample data event.
\nCan we find things outside of Flash that will let us do similar things?
\nThe answer is....of course! People have been writing desktop audio and music synthesizer software for years. Flash and AIR offers some decent functionality on the desktop, but writing something in C++ would blow Flash away of course.
\nThe reason is that Flash has only been in the generative audio game for a few years, while people writing C++ have been at this for decades. As such, they have access to ready made filters, libraries, etc.
\nTake, for example, Node Beat - written with OpenFrameworks
\nhttp://forum.openframeworks.cc/index.php?topic=6070.0
\n\nObjective-C for doing iPhone development has the nice benefit of access to core libraries that Apple OSX Cocoa developers have had access to for years. The basic AudioUnit in Cocoa or Objective-C gives you the low level access you need to make things like Mobile Synth:
\nhttp://code.google.com/p/mobilesynth/
\nOr even a simple tone generator:
\nhttp://cocoawithlove.com/2010/10/ios-tone-generator-introduction-to.html
\nBasically though, you'll find that anything that gives you low-level audio access will give you what you need. Speed is the key here, number crunching needs to be done in an instant. Flash can keep up OK, and gives you the bare minimum of what you'd need to generate sound.
\nWhat about Javascript? Well, sort of! The Web Audio API has been defined and implemented on the developer version of Chrome and there's also a Audio Data API draft in Firefox. The basics are the same as Flash, but many more goodies are offered to help you along.
\n\nSo, in the end we need speed and some basic API hooks. On the web, Flash seems the most reliable, but it sounds like Chrome and Firefox are nipping at Flash's heels with some very cool stuff coming up!
\nAnd in the mobile world, just like the desktop world, it seems this sort of base level audio access is standard as well.
\n", "date_published": "2011-09-04T00:00:00Z" } , { "id": "https://benfarrell.com/blog/2011-09-03-live-instrumentation-in-flash-voice-synthesis/", "url": "https://benfarrell.com/blog/2011-09-03-live-instrumentation-in-flash-voice-synthesis/", "title": "Live Instrumentation in Flash Part 5 - Voice Synthesis", "content_html": "So far, we've discussed how to create tones, notes, and chords. We've also gone into several different algorithms to change how a digital instrument sounds. Changing how our digital instrument sounds is a lot more than simply changing the algorithm that defines your sound wave! You may have noticed that given the several algorithms we covered in part 2 of this series, the sounds didn't change all that much.
\nWhat can we do to create some more variety?
\nAttack, Decay, Sustain, Release
\nIn electronic synthesizers, there are the concepts of "attack", "decay", "sustain", and "release" (commonly abbreviated ADSR). The easiest way to explain this is to imagine yourself hitting a key on a piano.
\nThe moment you hit the key, the string is hit by the hammer by the piano. This initially creates a loud tone. The initial tone is much louder than anything that happens moments after. This initial loudness is called the "attack". As the initial loudness wears off, the sound draws down to the normal volume as you hold the piano key. This drawdown is known as the "decay", and for the rest of the duration that you're holding the key is known as the sustain.
\nOnce you release that piano key (on a real piano), there would be a small amount of sound coming out as the string is still vibrating. This will eventually fade away - but this period is known as the "release" period.
\n![\"ADSR_inverted_parameter\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2011/09/ADSR_inverted_parameter.png\")
Envelopes
\nAttack, Decay, Sustain, Release is a type of envelope. At a basic level, an envelope is controls volume on a waveform as the cycles continue. There are more types of envelopes, but the ADSR envelope is the most commonly known.
\nSome synthesizers break up the phases even more. Consider the DAHDSR envelope. This starts with a delay phase. This is the phase before the attack - a phase where you don't hear any audio, or much at all, in the initial lead up to the loudness that is the attack phase.
\nThen comes the "hold" phase. This is meant to prolong the time between the attack and release. It keeps the volume high!
\nAnd then the rest continues with the decay, sustain, and release.
\nDifferent types of synthesizers take different envelope approaches. Some even let you create your own - however it seems that the simple ADSR is the most common.
\nShortening the Sustain
\nOne thing I've found is that a sustain that keeps going and going and going while you press a key, sounds kind of unnatural. It might be appropriate for something like a pipe organ, but if you're trying to replicate the percussiveness of a piano or a bass guitar pluck, you don't want something that goes on forever.
\nKeeping it short and sweet is the perfect key to making something sound natural! I'm still experimenting with envelopes to try to accomplish this effect without just cutting off the waveform. Overall though, I've found that limiting a sustain to a few hundred milliseconds is the perfect solution to making a more natural sounding instrument - especially if you want something percussive like a bass pluck or piano key.
\nHarmonic Overtones and other Imperfections
\nImperfections in the note quality are a good way to make a computer generated note sound more natural. Perhaps the tone could be off a couple steps in the frequency, however there are other ways!
\nOne such way is to give a note a harmonic overtone. Remember back in part 4 when we went into note relations? We talked about octaves, where an octave higher will produce the same key/tone, but is at a higher pitch. An octave higher or lower is in perfect harmony with the root note.
\nWith harmonic overtones, we layer several octaves on top of the note. This gives a more natural sound. Many instruments in the real world will have the first 2 overtones be in perfect harmony. In this regard, starting at middle A of 440hz, we'd have overtones of 880hz, 1320hz, and 1760hz. All these notes playing at the same time to produce something more natural!
\nAnother factor in real world instruments is that these overtones (especially the third and fourth) may not be exactly the same key in an octave, not in absolute harmony. These higher overtones might be a little sharp or a little flat. These nuances contribute to the instruments unique sound. In fact, if you've ever seen the flare on the end of the trumpet, it's not to get more volume, but to correct the harmonic overtones to be closer to absolute harmony.
\nWe can absolutely replicate this in our digital instruments! We can mix harmonic overtones together when playing a single note, and even make variations to avoid absolute harmony and give our instruments some character.
\nFilters, Modulations, and more
\nWe've only scratched the surface of creating new and exciting voices for our instruments. We can pass our tone through a filter, do frequency modulation, and more. I'm still exploring, so this is best left for another time!
\n", "date_published": "2011-09-03T00:00:00Z" } , { "id": "https://benfarrell.com/blog/2011-09-01-live-instrumentation-in-flash-part-4-a-little-music-theory/", "url": "https://benfarrell.com/blog/2011-09-01-live-instrumentation-in-flash-part-4-a-little-music-theory/", "title": "Live Instrumentation in Flash Part 4 - A Little Music Theory", "content_html": "In the last post, I threw a number at you: 440. This was the frequency I put in my code to make the resulting audio sample listenable.
\nWell, it was no accident. 440 cycles is actually a middle A.
\nA Brief History
\nIt wasn't always 440hz (hertz actually denotes cycles per second), but not for lack of trying! Remember that we haven't gone digital until the past 30 years or so, and industrial manufacturing is only 70-80 years old. People have been making instruments and playing music on them for thousands of years. Beethoven was playing the piano in the late 18th century.
\nWhen you hear Beethoven's music recreated, you're not hearing it exactly as he played it. Yes, all the music is transcribed and is probably fairly true to the original. However, tuning a piano in the 18th century wasn't an exact science. Beethoven's middle A might not have exactly been 440hz.
\nWhile Beethoven was a German composer, Chopin was Polish. Being in different countries, so far apart, Chopin probably had a different tuning than Beethoven. If you were in the German audience listening to Beethoven, you might be accustomed to a middle A tuned around 338hz. Whereas, if you traveled to Poland, and heard the 445hz middle A, you might think something was a bit off. You weren't used to the different tuning.
\nRecorded music and touring artists changed all this. As records were being sold and artists toured, someone in Japan could listen to a record produced in Spain. As folks started getting together from all over the world, a sort of standard tuning was made out of all of this. When digital music and exact measurements of frequencies came along, we could finally settle on a specific standard that everyone could agree on.
\n\nThe Math of Notes and Octaves
\nSo far, we've only talked about middle A, or "A4". Since A4 is basically the center of it all, we can go from there. First off, we can talk octaves. Take our middle A, for example. If you go down an octave, you land on another A....A3. Basically, its the same tone but a lower frequency.
\nHow, do we figure out the frequency? Well this time its easy - no fancy trigonometry! Just half the 440. So A3 is 220. A2 is 110. A5, on the other hand, is double 440 (880hz).
\nNow, what's between A4 and A5? What's between an octave? Well, there are 12 notes between the two. In music terminology going up 1 note is called a "half step". Going up 2 notes is called a "whole step".
\nStarting at A4 (or any A):
\nIf we assume that we're starting at A4, and going up 1/12 of an octave for each half step, there's actually a mathematical formula to calculate the frequency!
\nThat formula is: frequency = 440 * (Math.pow(2, indx/12));
\nYou should ALSO note that the relation between half-steps. If note X is a half step below note Y, note X is said to be Y "flat". Note Y is said to be X sharp. This applies to any key (not just the black keys on the piano are sharps and flats, though it makes total sense to name them that way given their positions).
\nSo in the Flashamaphone framework, I've gone ahead and started at middle A. I've put each of the twelve notes into an array. Whichever index we're trying to access in represented by "indx" in the above formula. We subtract 12 from index for each octave below 4, and add twelve for each octave above 4. What we have here, is a simple way to calculate frequency for a given note! Pretty cool!
\nHow Western music is different from others
\nJjust like how Beethoven's piano was tuned differently from Chopin's, different cultures have different connotations for chords. The most basic example is major vs minor. In our western culture, we associate major keys with a happy sound, but minor keys are sad. In other cultures, this doesn't necessarily hold true. In fact, Indian music has no concept of major vs minor....and so, no similar connotations.
\nWhat we'll cover next with chord structures will be accurate for any culture. However, what each chord construct invokes for a person in western civilization is completely different for other cultures. We've already discussed that major vs minor doesn't mean anything important for Indian music - but that doesn't mean the construct doesn't exist! It's all just different ways of describing the ways notes can make up a chord.
\nChords Structures
\nStarting from the root, you can get all the notes in a "key signature" the same way from any root note. A key signature groups notes together - and typically if you play only the notes in the key, your music won't sound off-putting.
\nA major key is always comprised of the first, third, fifth, seventh, ninth, and eleventh half tones up from your root.
\nA minor key is used in western culture to make something sound sad or similar. Likewise, its always comprised of the first, third, fourth, sixth, eighth, ninth, and eleventh half tones up from the root.
\nI take care of this in Flashamaphone by making the following arrays for you:
\n// major key notesInKey.push( notesToIndex[0] ); notesInKey.push( notesToIndex[2] ); notesInKey.push( notesToIndex[4] ); notesInKey.push( notesToIndex[5] ); notesInKey.push( notesToIndex[7] ); notesInKey.push( notesToIndex[9] ); notesInKey.push( notesToIndex[11] )
// minor key notesInKey.push( notesToIndex[0] ); notesInKey.push( notesToIndex[2] ); notesInKey.push( notesToIndex[3] ); notesInKey.push( notesToIndex[5] ); notesInKey.push( notesToIndex[7] ); notesInKey.push( notesToIndex[8] ); notesInKey.push( notesToIndex[10] );
The most common chord structure is a triad - this means that there are 3 notes in a chord. The first note is a chord is the "root". A "C" chord will always have the "C" note as the root, in other words.
\nA typical triad would consist of the 1st, 3rd, and 5th notes in the key signature. There's lots more chord types, which you can use Flashamaphone to generate for you, but that's the basics!
\n", "date_published": "2011-09-01T00:00:00Z" } , { "id": "https://benfarrell.com/blog/2011-08-30-live-instrumentation-in-flash-part-3-generating-a-tone/", "url": "https://benfarrell.com/blog/2011-08-30-live-instrumentation-in-flash-part-3-generating-a-tone/", "title": "Live Instrumentation in Flash Part 3 - Generating a Tone", "content_html": "Basic Sine Wave Generation
\nOK! Lets step into the wayback machine and go back to part 1 of this series. We were talking about building audio samples with a byte array
\nvar bytes:ByteArray = new ByteArray(); for (var c:int = 0; c < 8192; c++) { bytes.readFloat(number); bytes.readFloat(number); }
I never DID tell you guys what goes in number, did I?
\nWell, that's where we can start making something listenable (and not the white noise we made before).
\nLets make some changes:
\nfor(var c:int = 0; c < durationinseconds*44100; c++) { var number:Number = Math.sin(c * 2*Math.PI/44100 * 440); bytes.readFloat(number); bytes.readFloat(number); }
OK, so what's going on here? We've introduced, first of all, "durationinseconds". We multiply by 44100. Since there are 44,100 samples per second, all we need to do is multiply by the number of seconds we want to produce, and generate that many samples.
\nNext up is to generate a sine wave. Why a sine wave? Well, think about your high school algebra class. There are all sorts of mathematic equations, and you can graph any of them. Most basic is a simple line.
\n![\"linear\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2011/08/linear1.jpg\")
\na line - it starts infinitely low, and ends infinitely high, never repeating
The problem with a line, is that it keeps going up and up and up and up. That's really not what we want here. What we want is something that goes high, comes back down, goes up again, and can do that forever.
\nWe COULD utilize a little programming, and make a line, but at a set high point, reset and make it start at the low point again.
\n![\"sawtooth\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2011/08/sawtooth.jpg\")
\na line that cycles from going up and up, but resetting at 0 again
This is actually a good way to produce a sound, but lets talk sine waves for now.
\nThe sine function to produce wave cycles is fantastic. The line can go up and down all day as you continually increment, but not only that, it's curvy! So it goes up and down in a very smooth way.
\n![\"sinwave\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2011/08/sinwave.jpg\")
\nGoes up and down repeatedly (forever) and is curvy
The thing about sine waves is that since they repeat themselves, they can ALSO be measured like a circle - in degrees. If you go 360 degrees, you're right back where you started from. The bummer about trigonometry though, is that everything is measured in radians.
\nSo 360 actually is equal to 2 Pi. 180 is Pi, and 90 is Pi/2. So above, we took 2 Pi and divided by 44100. We're basically making a baseline here and saying that one full revolution through a sine wave is equivalent to one second. We multiply this by our iterator (c), to make the individual samples for each data point.
\nThis would actually produce a SUPER low tone (I doubt you'd be able to hear it). I'm going to go ahead and ALSO multiply by 440 to produce something listenable.
\nFrequency and Amplitude
\nSo let's talk about what we did when we multiplied by 440. The effect we had was to take that super low tone (the one that takes a whole second to go a complete cycle) and make it go 440 cycles in one second.
\nHow often something cycles is known as frequency. And that's the thing about frequency in sound. If the frequency is too low, that is it doesn't cycle fast enough, our brain doesn't put together that its a repeating pattern, and it can't latch on to the fact that its an audible tone. If too high, our ears can't actually discern the signals either.
\n440 cycles is pretty much smack in the middle of what we are comfortable hearing. There's a LITTLE more to this, and I'll tell you more in part 4.
\nAnyway, all those cycles going so fast come together sounding like one tone. The more cycles per second, the higher the tone.
\nWhen you've heard about frequency, you've also heard about amplitude. In terms of audio, amplitude is just volume. It's pretty easy. If you want a volume that's 1/4 as loud, just multiply that "number" variable by .25 (OK I'm lying, volume is actually logarithmic, but I don't feel like getting into that whole thing right now).
\nHere's a nice little demo to allow you to play around with frequency and amplitude on a sin wave.
\nDifferent Types of Cycles
\nLike we discussed, sine waves are curvy... When we generate the tone, its sounds nice (if not a little whiny). There are other ways to go. I eluded to this before (with the line that we keep resetting in a cycle). You can make things sound edgy too! Like you could make a square wave. I'm going to start giving examples now, and copy over some of what I have in the Flashamaphone project.
\nFirst of all though, lets simplify things and pop a bunch of math into a phase variable, like so:
\nvar phase = c * 2*Math.PI/44100 * 440;
There! Now we don't have to write all that stuff out each time. So let's revisit how to do a sine wave:
\n// loop number = Math.sin(phase); // end loop
\nGoes up and down repeatedly (forever) and is curvy
Next, lets try a square wave
\n// loop number = Math.floor(Math.sin(phase)); // end loop
![\"squarewave\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2011/08/squarewave.jpg\")
It sounds very 8-bit and harsh.
\nAt this point, I can start making up my own terminology - and I came up with a stepped wave. It's half-way between harsh and smooth, between a sine wave and a square wave.
\n// loop number = Math.floor(Math.sin(phase)*4)/8 - Math.floor(Math.cos(phase)*4)/8; // end loop
![\"steppedsinewave\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2011/08/steppedsinewave.jpg\")
I have more!
\nA Step wave? // loop number = Math.floor(Math.sin(phase)) - Math.floor(Math.cos(phase)); // end loop
![\"stepwave\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2011/08/stepwave.jpg\")
Shark fin? // loop number = Math.cos(phase) - Math.floor(Math.sin(phase)); // end loop
![\"sharkfin\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2011/08/sharkfin.jpg\")
Saw tooth? // loop number = phase - Math.floor(phase); // end loop
\na line that cycles from going up and up, but resetting at 0 again
Saw Sine? // loop number = Math.sin(phase) - Math.floor(Math.sin(phase)); // end loop
![\"sawsine\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2011/08/sawsine.jpg\")
To listen to these examples, check out this demo!
\nThat's all I have in the Flashamaphone project. We'll see soon in part 5 of this series that there is much more to producing different types of sound than just the math to generate the cycle. First though, jump on over to part 4, and we'll talk music theory!
\n", "date_published": "2011-08-30T00:00:00Z" } , { "id": "https://benfarrell.com/blog/2011-08-29-live-instrumentation-in-flash/", "url": "https://benfarrell.com/blog/2011-08-29-live-instrumentation-in-flash/", "title": "Live Instrumentation in Flash", "content_html": "This post covers another topic that I'll be covering at NCDevCon (http://ncdevcon.com/). Like my last post, I'm giving a general outline of series of points I'll be covering in both the presentation and the blog post and filling them in with links as I progress.
\nLive Instrumentation is a topic I'm an amateur in, but is terribly interesting to me. It makes me remember back in the 80's playing with synthesizers as a kid, and tweaking knobs to make different sounds. That is, until they got complicated and I sorta gave up. Also, samplers came along and made things a little less relevant.
\nAnyway - my Live Instrumentation series will cover how to create sounds dynamically in Flash. We'll explore how these sounds can be made into musical notes, chords, compositions, etc. We'll also explore some synthesizer basics - that is making our notes have different voices, or different sound qualities.
\n\nBasics of Dynamic Sound in Flash
\nAs I said in part 1 of this Live Instrumentation series, working with byte arrays can be hard, and take some getting used to. I still stand by the statement that you shouldn't be afraid though! One of the best ways to get cracking is to get your hands on an existing project and start changing stuff. See what works and what doesn't work.
\nI don't know of too many projects that deal with audio generation. Probably the best one out there is called Tonfall (http://code.google.com/p/tonfall/). Tonfall brands itself as a Tiny AS3 Audio Framework. It has lots of cool stuff to get you going playing with audio. What I didn't like about it for my purposes was the learning curve, especially if you don't know much about audio already. They have some cool demos - but when I was getting started in this stuff, the first thing that popped into my mind was a piano keyboard. With Tonfall, I didn't see a way to easily play a specific note like an A sharp. Instead, I'd have to know the frequency of the note.
\nSo that's where I went when I started up my own project called Flashamaphone! I wanted a way to easily make notes and chords and play them live. I wanted a way to get started if you only new a tiny bit of Flex/AS3 and a tiny bit of music knowledge (only knowing your way around piano keys a bit).
\nLets rehash a little bit of what we already covered as it relates to Flashamaphone. First off, I've included the concept of a buffer in Flashamaphone. Currently there are two types of audio buffers in the project
\nRecorded Buffer
\nThis is the simplest of the buffers. Its only does a few things. You can add bytes to the buffer with "addToBuffer", you can get the buffer bytes with "get buffer", clear the buffer with "clearBuffer", or activate or pull from a controller. We'll get into the controller in a bit....
\nSo what you end up with here, is basically an audio file in memory that you can write to and read from. Pretty simple!
\nLive Buffer
\nThe live buffer is a little bit more complicated, but same concepts. You can add bytes to the buffer, but that's not really the point here. The real point is to pull from a controller (wait one more second for that explanation!). The real mechanical stuff going on here is a sound object with a sample data event. Every time the audi0 buffer reaches out and needs more data, we'll pull a certain number of samples from our controller. If a controller doesn't exist, we'll pull it from a queue of audio bytes.
\nPulling data when we need it (and only when we need it), means that we are creating a sort of live audio playback scenario.
\nKeyboard Controller
\nSo far, there's just one type of controller in the Flashamaphone project. Its the keyboard controller. The keyboard controller allows you to "press" a key and "release". With each press and release, you pass in the tone or note that you're pressing. In the end, all this controller does is keep track of what keys are currently being pressed, and what keys are in a "released" mode as the sound fades away after it's been released.
\nNow, when we attach a controller to the buffer (whether live or recorded), the buffer can pull from the controller. The act of pulling from the controller means that the buffer will ask for the bytes of a specific number of samples from the controller. If the "A", "D", and "D#" keys are pressed on the controller, the controller will send back a mix of those bytes. The buffer doesn't care what keys are pressed, it just wants whatever the bytes are.
\nYou've been Caught Up
\nRight, so you've been caught up on the "engine" behind Flashamaphone. That's all it really is behind the scenes. A couple buffers and a controller. It's not very fun to just push bytes back and forth, but it needed to be done to get to the real cool stuff. Next we'll be making some actual recognizable tones!
\nFirst, let me hit you with a Flex code sample of our white noise using the keycontroller and live buffer using Flashamaphone:
\nThe code above simply creates both a live buffer and a keyboard controller. It assigns the live buffer to the keyboard controller. Now, all we need to figure out is how to send some actual notes to our keyboard controller, and not just white noise!
\n", "date_published": "2011-08-29T00:00:00Z" } , { "id": "https://benfarrell.com/blog/2011-08-29-live-instrumentation-in-flash-part-1-basics-of-dynamic-sound/", "url": "https://benfarrell.com/blog/2011-08-29-live-instrumentation-in-flash-part-1-basics-of-dynamic-sound/", "title": "Live Instrumentation in Flash Part 1 - Basics of Dynamic Sound", "content_html": "Lets start with the basics. We need to create sound from thin air and write that sound somewhere so we can hear it. This could be live as you press a button, or into a file so you can play it later.
\n\nDon't Be Afraid of the Byte Array!
\nIn Flash, or most platforms in fact, there are a lot of different data types. You have integers, floats, strings, etc. I can make a sentence by putting strings together, and I can make an array by putting various types of data together. A byte array is like an array, but it's a long concatenation of binary data.
\nUnfortunately byteArrays are hard to debug! You can't really just trace stuff out to your console like with a string, since it's not in a readable format anymore.
\nHow do you make binary data? Well it's easy with the Flash APIs. You can read a string into a byte array - or more appropriate to us for sound: copy and paste bytes into a byte array or read a floating point number into a byte array.
\nWhen specifically talking about creating sound, we'll mostly be dealing with reading floats into a byte array many times over in a loop.
\nFor example, if you wanted to create a 4 second sound, there is some math to figure out. Lets start with a loop:
\nvar bytes:ByteArray = new ByteArray(); for (var c:int = 0; c < len; c++) { bytes.writeFloat(number); }
So what is "len"? How many times should we iterate and add floats into our byte array? Well typically, we're dealing with 44,100 samples per second. That means for every second of audio we're using 44,100 data points if you picture our sound like a line graph.
\nOur code snippet becomes:
\nvar bytes:ByteArray = new ByteArray(); for (var c:int = 0; c < 4 * 44100; c++) { bytes.writeFloat(number); }
One more thing, though. The code above assumes mono, or one channel sound. If we want stereo sound, we need to do this loop twice over. Unless you're trying to accomplish different effects on the left and right side, it's usually fair to say that the float you read in each loop iteration will be the same number. This is why you'll usually see the readFloat line repeated instead of the iteration count going up to 88,200:
\nvar bytes:ByteArray = new ByteArray(); for (var c:int = 0; c < 4 * 44100; c++) { bytes.writeFloat(number); bytes.writeFloat(number); }
What "number" are we writing? Well....thats where the magic happens, and is a bit more complicated. Don't worry, we'll get to that later.
\nWriting File Output Having a byte array in memory is all well and good, but how can we use this. In terms of audio, we can dump it to a file, or play it live.
\nDumping to a file is pretty easy if you find a good encoder. MP3 is compact, but processor intensive. It's also kinda hard to figure out if you're starting from scratch. Long story short, you're writing chunks of data (frames) and giving each frame a header describing the contents of each data frame. You're also compacting the data somehow. It's a little beyond me, which is why if I ever need to do it, I'd grab a fast, ready made library like Shine (http://code.google.com/p/flash-kikko/). Shine is made in Alchemy - which means someone wrote it in C++ and compiled the library to a Flash SWC file. Doing it this way can result in much faster number crunching!
\nBut, I like to be a little more basic - and write to a WAV file. WAV files are a format that contains raw audio data. For the most part we can take our generated audio data, and write it as is! Just dump all those bytes into a file!
\nUnfortunately, even writing audio to a WAV format takes some know how. You need to construct a file header which includes the number of channels, sampling rate, and a whole bunch of other things. Once all that stuff gets written to the file, you can just dump your actual audio data.
\nI'm not a WAV file format scholar, or do I really even understand whats going on in the header. This is why I'm using the WAV audio encoder from the Tonfall project by Andre Michelle (http://code.google.com/p/tonfall/).
\nHere's an example of writing to a file on my desktop with AIR, but passing it through Tonfall's WAV encoder first.
\nvar we:WAVEncoder = new WAVEncoder(WAV16BitStereo44Khz.INSTANCE); we.write32BitStereo44KHz(myaudiobytes.bytes, myaudiobytes.bytes.length/8); var file:File = File.desktopDirectory.resolvePath("output/song.wav"); var stream:FileStream = new FileStream(); stream.open(file, FileMode.WRITE); stream.writeBytes(we.bytes); stream.close();
Did you notice the line where we're passing myaudiobytes.bytes.length/8? Well this parameter is how many "samples" we're passing. It leads to an important fun fact. A floating point number is 4 bytes. If we're talking a mono file, a sample would be one floating point number or the 4 bytes.
\nSince we're talking about a STEREO file, it's 4 times 2, or 8 bytes. So to figure out how many stereo samples are in a byte array, we'd just take the length of the array and divide by 8.
\nWhat if we wanted to know how many SECONDS of audio are in our byte array? Well, first we'd find the number of samples, but then divide THAT by the sample rate (44,100 samples per second).
\nLive Playback with an Audio Buffer
\nI'm glad we talked about dumping stuff into a file first, because basically using an audio buffer is just like that, but you do it in smaller chunks.
\nTo get the ball rolling, what you'd do is start up a blank sound in Flash. Usually you give it a source to play, but not here! Here, just a blank sound. Next, add an event listener to the sound "SAMPLE_DATA". Whenever the audio buffer runs out of stuff to play, it will fire off the sample data event. Your code will pick up the event and run a function. Ideally, in this function, you'd be putting stuff into the audio buffer.
\nAfter adding the callback, play the sound to get things started:
\nsound = new Sound(); sound.addEventListener(SampleDataEvent.SAMPLE_DATA, onSampleData, false, 0, true); sound.play();
What does our callback method look like? Well, for the most part, its what we discussed when popping stuff into a file. Just in smaller chunks.
\nfunction onSampleData(event):void { var bytes:ByteArray = new ByteArray(); for (var c:int = 0; c < 8192; c++) { bytes.writeFloat(number); bytes.writeFloat(number); } event.data.writeBytes(bytes); }
OK, so there's not much difference here! We're iterating over the same type of loop - writing stereo audio. We're writing 8192 samples though. This IS a magic number. It's the upper limit of what Flash allows you to write in the buffer. The lower limit is 2048. Now, why use one or the other (or in-between).
\nSo, if you use the lower limit of 2048, audio will be sampled more frequently. It will tax your processor more often - and it might not be able to keep up properly. This means that when you listen to the audio being generated there may be some clicks or pops in your headphones as audio drops out and comes back in.
\nIf you use the maximum limit of 8192, your processor is hit less often - and when it is hit, it can almost always handle the sample data generation. So thats good! Problem is that all this data, all 8192 samples represent a good quarter of a second or so of audio. This means that your audio could be lagging by a quarter of a second. I don't mean that audio won't play back smoothly. What I DO mean is that if you press a button, it could take you 1/4 of a second to hear the result of that button press. This is a little slow to make something like a virtual piano that feels reactive!
\nAt the very end though, you're writing to the event.data byte array. In fact, you don't even have to generate your own sound. You could just be reading slowly from a byte array of a raw audio file you already read in. In fact, you could run some mathematical filter on it, and alter the values. Folks have done this before by loading an MP3 file into the sound object, using sound.extract to load it into a byte array, and THEN dropping every other sample as it gets sent to the buffer to do something like increasing the speed by 2x.
\nWhite Noise I told you we'd get into what the number value is when we read the float into the byte array. That time is not yet - it'll come in part 3. However, lets just put some random values in there!
\nfunction onSampleData(event):void { var bytes:ByteArray = new ByteArray(); for (var c:int = 0; c < 8192; c++) { var number:Number = Math.random(); bytes.writeFloat(number); bytes.writeFloat(number); } event.data.writeBytes(bytes); }
By adding random values, we've just added random samples....at no real discernable frequency. This means there's tons of noise, but no way to hear one strong tone from the noise. It results in white noise - like TV snow, or waves at the beach.
\nLet's take a peek at what this looks like when we visualize it:
\n![\"whitenoise\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2011/08/whitenoise.jpg\")
2024 Update: Boo! Flash doesn't work anymore on the web. Sorry for the empty link!
\nAnd lets listen to what it sounds like (Flash Example)
\n", "date_published": "2011-08-29T00:00:00Z" } , { "id": "https://benfarrell.com/blog/2011-08-28-mercury-charm-offensive-2/", "url": "https://benfarrell.com/blog/2011-08-28-mercury-charm-offensive-2/", "title": "Mercury Charm Offensive", "content_html": "Just a heads up that my old band has a facebook page now, we have a buncha videos and songs over at http://www.facebook.com/pages/Mercury-Charm-Offensive/238338422876882
\n", "date_published": "2011-08-28T00:00:00Z" } , { "id": "https://benfarrell.com/blog/2011-08-28-html5-vs-flash-video-choose-wisely/", "url": "https://benfarrell.com/blog/2011-08-28-html5-vs-flash-video-choose-wisely/", "title": "HTML5 vs Flash Video: Choose Wisely", "content_html": "The title of this blog post is ALSO the title of an upcoming talk I'm giving at NCDevCon (http://ncdevcon.com/). I've decided that instead of starting out in power point, I'd write this up as a blog post and think about how to present this stuff on a big screen later. I do this, not only to have a nice blog post about this stuff, but because I hate looking at power presos on my computer - and I'd rather have something better to share with folks after the presentation.
\nI'll write this up as a series of posts with this as the main index page. So below are the main topics in this series! I'll fill them in with links as I progress, so stay tuned!
\n\n\nHow Apple HTML5 is Different than normal HTML5
\nWhat's Happening on the Server Side
\nIn addition to running two shows on Codebass Radio (one music, one tech talk), I'm on the management team. I've helped out with graphics here and there, so of course when station owner Vicky Ryder wanted a logo, I was happy to help!
\nHere is the styled final logo
\n![\"codebass_radio_color\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2011/08/codebass_radio_color.jpg\")
And here it is on a tshirt!
\n![\"tee\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2011/08/tee.jpg\")
Last, a black and white version:
\n![\"codebass_radio_B&W\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2011/08/codebass_radio_BW.png\")
My wife runs a restaurant and cocktail review website called "The Gourmez". I designed a website for her and built a WordPress style around the design. Shown here is the second revision I made for her in 2010.
\nClick the image to go to the site ![\"the](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2011/08/gourmez.jpg\")
Here's the old version of the site, prior to 2010:![\"Gourmez](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2011/08/Gourmez-Website.jpg\")
Runtime Expectations is a live talk show on software development. I co-host with Bucky Schwarz and Adrian Pomilio.
\nThe show page is over on CodebassRadio:
\nhttp://codebassradio.net/shows/runtime-expectations/
\nWe also have a meetup page that we schedule shows and invite people to sit in: http://runtimeexpectations.com.
\nAdrian came up with the show name - I warmed up to it pretty fast despite wanting to come up with something better. Now that we're 8shows in, I couldn't imagine a better name.
\nHere's the final logo I came up with. I originally thought we could base it on the Windows alert when an error pops up. Adrian had the idea to tilt it, and then I put the finishing touches on.
\n![\"rte_black\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2011/08/rte_black.jpg\")
Here's what we had for the first couple of shows. The Durham American Tobacco District is pictured in the background cause that's where we broadcast from
\n![\"rte\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2011/08/rte.jpg\")
NCDevCon is an awesome annual conference in Raleigh, NC. It focuses on Flash, Flex, Coldfusion, Web, and Mobile.
\nThe organizers had asked me to help design a logo. I came up with a black and white concept for them, and ended up stopping there since they wanted a simple black and white webpage and something to print on t-shirts and marketing material.
\n![\"ncdevcon-logo\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2011/08/ncdevcon-logo.png\"){kind=logo}
It came out extremely simply, but I think everyone likes it. Its interesting to hear people's takes on what the logo symbolizes. With so many technologies in one conference, how to symbolize them all? I chose to go for something that symbolized an open, new document. Like the icon you see when choosing blank/new document from a menu (where the corner of the page is turned in).
\nAnyway, I did that - but I heard that it did a good job to represent the Raleigh skyline. I think the best logos are simple and can be interpreted however the viewer sees them.
\nBelow are some more sketches I did before landing on the final
\n![\"ncdevcon\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2011/08/ncdevcon1.jpg\")
My wife, author of "Maya's Vacation" needed to send a bookmark to press in order to advertise her book. I whipped up something in a few hours. Note that the art is not mine, but rather drawn by an author at the publishing company
\n![\"bookmark\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2011/08/bookmark.jpg\")
So, my brother in law needed a website. He sells Amsoil and wanted to sell it through his website as an affiliate.
\nI helped him get a site off the ground and designed. I did his logo as well. To be honest, this was my first experience telling\nsomeone "yes" whenever they wanted something. For design sensibilities, it wasn't a great idea - but being free, I didn't want to\nplan and handhold as I would a professional client.
\n![\"LubeHelp\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2011/08/LubeHelp.jpg\")
I was just going back through the archives and digging up screenshots of old portfolio sites and CD's. Kind of embarrassing how bad things looked all those years ago, but it was in 1997, who can blame me!
\nMy 1997 website hosted on my school's webserver!
\n![\"1997\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2011/08/1997.jpg\")
1998 - Same deal....still looks pretty bad even with the glassy effect!
\n![\"1998\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2011/08/1998.jpg\")
A portfolio CD from 2000! I toted this one with me around NY and Boston for a few interviews as I was fresh outta college.
\n![\"2000\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2011/08/20001.jpg\")
My year 2000 website - interactive breakfast themed!
\n![\"2000Website\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2011/08/2000Website.jpg\")
I don't even know what this was....no time? I had an idea for Yellow5Labs in 2003 - a take on the additive food dye. I killed it off, cause people thought I was really into labradore retrievers. But not before going through a few more iterations....![\"2003Website\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2011/08/2003Website.jpg\")
In 2005, I seemed to get my act together to make something good looking. The chemical compound flies across the screen with your mouse. Hovering over it will zoom into the hexagon and load up a section of the site on the right![\"2005website\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2011/08/2005website.jpg\")
This is the final iteration of Yellow5Labs from 2008 or 2009. As I said, I killed it off cause people thought I was into labradore retrievers. I couldn't find a screenshot, but here it is as a comp without copy. The test tubes blow up as you mouse over (shooting up vertically). And the cloud is the header, while the lab technician is the footer, with the part in-between expanding vertically with the copy.![\"2009website\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2011/08/2009website.jpg\")
Some time ago, I joined a "Subversive Humor" meetup. I think that's what it was. Anyway, the organizer and I started working on a comic strip about a web startup company that went under. The owner of the company inherited a tobacco plantation in the south, so picked up and moved from NYC to NC.
\nI started with some pencil sketches, and eventually brought them into Adobe Illustrator as 2D line drawings. I eventually fleshed them out to a more 3D look using gradient meshes.
\n![\"brett_cs4\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2011/08/brett.jpg\")
![\"clayton_cs3\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2011/08/clayton_cs3.jpg\")
Back when I ran the Raleigh Durham Adobe User Group, I was a one man show. I thought it would be a great idea to join forces with Adrian Pomilio who ran the local Flex User Group. We came up with the name "Tri-Me" for Triangle Media Experience.
\nI came up with some logo concepts:
\n![\"trime\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2011/08/trime.jpg\")
And then I comped up a webpage. Adrian thought it was really ugly with the colors. I actually still dig it. Needless to say it was scrapped:
\n![\"tri-me_comp\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2011/08/tri-me_comp.jpg\")
My friend Jason Burke had asked me to help him on his children's show spoof he was working on for his senior thesis at Emerson. I was asked to draw some scenes that he would composite live actors over
\nThe show took place in a treehouse - the exterior here:
\n![\"chuckles_treehouse\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2011/08/chuckles_treehouse.jpg\")
And the interior here:
\n![\"chuckles_treehouseinside\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2011/08/chuckles_treehouseinside.jpg\")
And yah, occasionally I'm not the best with perspective, but its a kids show, so I ran with the look!
\nJason wanted to hit the whole consumerism angle, so we have "Chuckles" brand action figures on display:
\n![\"chuckles_toys_display\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2011/08/chuckles_toys_display.jpg\")
A mall's pretzel stand:
\n![\"chuckles_pretzelshack\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2011/08/chuckles_pretzelshack.jpg\")
The mall itself:![\"chuckles_mall\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2011/08/chuckles_mall.jpg\")
"Chuckles" brand clothing:
\n![\"chuckles_clothing\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2011/08/chuckles_clothing.jpg\")
And finally - la piece de resistance, "Chuckles" kids working a sweatshop to make the clothes:
\n![\"chuckles_sewingmachines\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2011/08/chuckles_sewingmachines.jpg\")
2024 Update: Boo! Flash doesn't work anymore on the web. Sorry for the empty link!
\nA "Save the Date" for my wedding - I animated this in Flash to a wedding showtune. Click the image to view
I used to run the Raleigh Durham Adobe User Group. We joined forces with some other local groups when the Adobe team came around to showcase their new AIR runtime.
\nThis is a poster I designed to advertise the event
\n![\"airshow\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2011/08/airshow.jpg\")
A project still in progress. Blastanova aims to combine music and games. See http://blastanova.com for more details. (update: sorry, this site was taken down)
\nTo support the project, I created a website:
\n![\"bnova_webscreen\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2011/08/bnova_webscreen.jpg\")
And business card:![\"blastanovabusinesscard\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2011/08/blastanovabusinesscard.jpg\")
I also created and designed a music/beat scanning application in Flex:
\n![\"\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2011/08/sambaverse.jpg\" "\\"sambaverse\\"")
(update: sorry, this site was taken down)
I had a big problem today with backwards compatibility and the Flex 4.5 SDK today.
\nI know....GASP!
\nBut, next let me say that it didn't have anything to do with the Flex framework, it was an Actionscript 3 project. THAT surprised me.
\nSo what was it? Well, the crux of the issue is that if you use the embed directive to compile in a visual element in your project, said project will completely fail in Flash 10.0.
\nHere's what I mean:
\nCreate a new AS3 project. Don't make it do anything - just embed a graphic.
\npackage { import flash.display.Sprite; public class TestEmbed extends Sprite { [Embed(source="myimg.jpg")] public var myGraphic:Class; public function TestEmbed(){} } }
No - this code doesn't actually do anything. It just creates an empty flash project with your graphic compiled in. And it works just fine if you test it on your machine.
\nNow, take that SWF somewhere that you haven't upgraded to Flash 10.1 yet.
\nGuess what? The constructor doesn't even run - instead you get this error:
\nVerifyError: Error #1053: Illegal override of z in mx.core.BitmapAsset.
I tried the same with library graphics from a SWF - this time I get a problem with mx.core.SpriteAsset.
\nWhat is actually happening here?
\nWell, when you use the embed directive, your source file is actually cast to a BitmapAsset for use in your project. Likewise, your SWF symbol is cast to a SpriteAsset.
\nThese are the trouble classes. Its worth noting that if your project just calls new BitmapAsset(), you'll get the same problem.
\nA SoundAsset isn't a visual element. Lo and behold, instantiating one of those works just fine.
\nEventually I narrowed the problem down to playerglobal.swc. Swapping out the 10.0 playerglobal with the 10.2 playerglobal produces the same error.
\nLets add this up. It's a problem with display assets, and the actual error says there's an illegal override of Z. I'm guessing that the faux 3D that Adobe added in 10.0 where you can set the Z had an API change from the 10.0 player to the 10.2 player.
\nWhen you load up your 10.2 SWF in a 10.0 player, that Z getter/setter is so drastically different, Flash just can't continue.
\nSo - there it is. If you use the embed directive and compile in the 4.5 SDK, your SWF won't function in anything less than Flash 10.1
\n", "date_published": "2011-05-26T00:00:00Z" } , { "id": "https://benfarrell.com/blog/2011-04-11-cocoon-p2p/", "url": "https://benfarrell.com/blog/2011-04-11-cocoon-p2p/", "title": "Cocoon P2P", "content_html": "This is just a short post - mainly for myself to remind me of the opensource awesomeness that was just released. It's called Cocoon P2P.
\nWhat this project appears to be is a peer to peer helper application for Flash powered devices to communicate in the same network. So basically, if I have a Android phone, a laptop, and an Android tablet all running Flash on my wi-fi network, they can talk to each other - without a server!
\nIt sounds fantastic, and available at http://code.google.com/p/cocoon-p2p/. The source code doesn't look that hefty either, so yay for easy to understanding!
\n", "date_published": "2011-04-11T00:00:00Z" } , { "id": "https://benfarrell.com/blog/2011-01-15-use-flash-text-engine-in-mx-components-in-ant/", "url": "https://benfarrell.com/blog/2011-01-15-use-flash-text-engine-in-mx-components-in-ant/", "title": "Use Flash Text Engine in MX Components in ANT", "content_html": "The title of this post is what I googled endlessly for like 2 hours trying to figure out how to do this.
\nHopefully you'll come here when you google this and get the answer you need. For you fine folks, I'll save you some time. Add this to your compiler options:
\nFor the rest of you - maybe you'd like to know what this is or get a little more background.
\n"Use Flash Text Engine in MX Components" is a setting you can check in the Flash Builder compiler options for a project. Old style Halo (MX) components from Flex 3 use the old text engine.
\nAs of Flex 4, we have a brand new text engine. Unfortunately, Adobe hasn't completely written all the new components yet for Flex 4 (Spark). This means that your Flex 4 project will probably have a mix of Spark and Halo (MX) components.
\nOne of the most frustrating things about this mix for me is working with fonts. For Spark components, you need to embed your fonts in the Compact Font Format (embedAsCFF=true). For MX components, you need to embed your fonts as non CFF (embedAsCFF=false). This could potentially mean embedding one set of fonts for your MX components and another set for your Spark Components.
\nNot only does this increase your application size, but it can get really hairy trying to work with styles. You need to identify all components that use Halo/MX components and tell them to use the correct font. If you go low level like label, you need to recognize that things like Alerts are MX components and they use MX labels, while if you just make a Spark label, you'll be using Spark. So this means creating separate styles for s|Label and mx|Label.
\nFortunately, Adobe made a workaround in our MX components. They allow you to use the new text engine in your MX components by checking the "Use Flash Text Engine in MX Components" option in Flash Builder.
\n![\"useFTEinMX\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2011/useFTEinMX.jpg\")
While that's great in Flash Builder, what if you want to use this option outside of Flash Builder? What if you want to do a command line compile, or better yet create an ANT build file.
\nFor the command line option, information is relatively easy to find: http://help.adobe.com/en_US/flex/using/WSda78ed3a750d6b8f-26a13bbf123c441239d-8000.html
\nBasically just use the theme argument and add a theme to the compiler options -theme+=myflexhome/frameworks/projects/spark/MXFTEText.css
\nThis is a special CSS file that Adobe made just for this purpose.
\nI was a little stumped on how to put it in an ANT build script, though. I tried:
\nUnfortunately, this completely replaces the theme, and doesn't add a CSS file to your themes. It worked decently enough for my project (save all the warnings I got).
\nLuckily another guy on my project, TJ Downes figured out that if you use the "append=true" ANT tag option, you can make this property additive, so you end up with:
\nSo there you have it. I couldn't find this anywhere, so hope this helps you!
\n", "date_published": "2011-01-15T00:00:00Z" } , { "id": "https://benfarrell.com/blog/2011-01-12-be-a-subject-expert-in-something-else/", "url": "https://benfarrell.com/blog/2011-01-12-be-a-subject-expert-in-something-else/", "title": "Be a Subject Expert (in something else)", "content_html": "Next week, my user interface user group, tri-me.org will be having our first meeting of the year. Rachel Nabors will be presenting on Wabi-Sabi in web design. Many times, a user group (ourselves included) will have a very technical presentation. How do you make the best MVC database application? How do you do memory management in Flex?
\nBLAH BLAH BLAH
\nI haven't seen Rachel's presentation yet, but it got me thinking about how inwardly focused web designers, developers, application programmers, etc are with our craft. Rachel is breaking out of this, and I think it's a thing that some of us do to not go crazy - but I don't think we realize how professionally useful it can be to break out of this.
\nMany of my friends and peers can do some amazing things behind a keyboard - both artistically and technically. We stay appraised of what Apple did to Adobe, what Google did to Apple, what the newest web framework is, and why it sucks. Our personal projects range from innovative ways to make a content management system, access a database, or log memory management in a project. We compete with each other in good AND bad ways and aspire to be the smartest kid on the block.
\nThis is all great. We're "keeping up with our industry". We're chasing technology after technology and remaining marketable. I wouldn't have survived this long if I didn't love the thrill of doing something innovative with new tech.
\nBut guess what? We can't relate to anybody but ourselves. We're either magical in the ways we can work a computer, scam artists with how much we charge, or both to people outside our circle. This really isn't too much of a problem professionally if you're great at what you do, but how good is it really?
\nMy wife is a writer, and to be honest, the profession bores me. I don't care about grammatically correct, mapping plot points, sentence flow, etc. To her credit, though, she doesn't write long essays about grammar. She writes things that non-writers can relate to, and she uses the knowledge of her craft to reach out to an audience that has no idea how to write.
\nWords are a medium to be shaped to send a message, just like pixels are. But if all we know is our craft, what kind of message can we send? Do we create a message only our peer group can understand, or do we rely on other people to tell us what message to create?
\nWe all have hobbies - sometimes our hobbies are database programming. But other times, our hobbies are things that everyone can enjoy like wine, music, dining, television, or Japanese art history. Our hobbies can have a way of looping back into our professional lives and we suddenly have a message, and the knowhow of our craft to push those pixels and send that message.
\nSome examples....
\nMy ex-employer 360KID was founded by Scott Traylor. He was interested in kids, their play, and how to use toys for education. He also happened to be a pretty decent web designer. He could have stayed a web designer, and just took whatever work came his way. Instead, he focused on children's education and entertainment, because it was what he was interested in. Eventually 360KID was in the very unique position of being an award winning technology and design company with intimate knowledge of how to push those pixels so kids could learn. Since he was a subject expert (in something else), we were one of the best companies in the country to go to if you wanted to design something educational for kids.
\nTom McCay, a NC local who I've met on several occasions, works at a company called rPath. They create build systems for programmers like me. BUT, Tom is passionate and knowledgeable about wine and beer. This led him to create winebythebar.com - a unique rating, review, and organizational mobile tool for wine that you can use by scanning bar codes. Tom is a subject expert (in something else). He's a smart guy, and could get hired at any tech savvy software development company he wanted to work at. But on top of that his expertise make give him an extremely unique skillset that most likely only he possesses. He can go after these kinds of specific business opportunities (and no doubt win them) because he turned a hobby into something more.
\nMe, I like music. I've been in bands, play the keyboard, and listen to bands you've never heard of. I'm not particularly good at any of this, but I am a web developer as well. As a result, I'm now writing music apps, music tech articles, and doing cool stuff that nobody else does - because I've turned myself into a subject expert (in something else).
\nI think in the end, our unique qualities keep us sane and give us perspective. The more we keep ourselves pointed inward, the less unique we are. There's only so many directions we can go in our design/development fields, and we become worth a dime a dozen because there's hundreds of other people out there that can structure a database AND make cool icons.
\nThe farther we reach outside our professional zone, the better chance we have of being unique and gaining this perspective. Sometimes having this unique perspective is the THING that makes us marketable regardless if this unique perspective is in music, wine, or Japanese art.
\nSo my advice, for whatever it's worth, is to be a subject expert (in something else).
\n", "date_published": "2011-01-12T00:00:00Z" } , { "id": "https://benfarrell.com/blog/2010-08-18-sambaverse-alpha/", "url": "https://benfarrell.com/blog/2010-08-18-sambaverse-alpha/", "title": "Sambaverse Alpha", "content_html": "Finally! I'm at the point where I have a demo-able application. Sambaverse is the first phase of my master plan for Blastanova.
\n![\"sambaverse_beats\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2010/08/sambaverse_beats.jpg\")
Quite some time ago, I got interested in creating music based Flash games. I created a prototype, and it worked pretty well - but it was severely lacking in one aspect. That aspect was the ability to look ahead in music, to know what's coming up 5 seconds from now or even 5 milliseconds from now. Or to be able to look back at what happened in the music before. Or to be a little smarter about logical song breaks.
\nWhile Flash has some decent realtime audio processing capabilities, realtime just wasn't good enough - my music games needed mystical, physic powers...to be able to see into the future and know how the musician who composed the piece thinks.
\nIn real terms, I needed a tool that was smart enough to load an MP3, take a good stab at automatically detecting beats, break, loud sections, and different sections of the song - like verses and choruses.
\nSo, I set out to create Sambaverse. A person using this application loads up an MP3. Right away the gears start turning - an audio waveform and song navigator is brought up in a few seconds. You can browse around, and listen to snippets. You can properly visualize your song. That's not very special though....you can do that in any audio editor. This is why I made some music visualization modes...
\nusing the application At the upper right of the application is the "Playback Visualization" menu. You can choose one or multiple options here. You have the ability to view the song beats, look where the quiet periods of the song are, where the most intense portions of the song are, and find logical breaks (sections) in the music like where a chorus ends and the bridge begins.
\n![\"sambaverse_viewmodemenu\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2010/08/sambaverse_viewmodemenu.jpg\")
Unfortunately, automatically finding logical breaks in the song was a little tough, and it will never be perfect. I'd describe my break detection as a good start, I have quite a ways to go. This is why I've added two drop down options. The first is the "custom song sections". This mirrors the automatically detected segments, until you drag them around to place them where you want. You can fully customize and refine these segments.
\nLikewise, I added a song exclamation overlay. Using this mode will place little stars on the timeline. The purpose of these stars is to signify special musical events that occur very rarely. Like when that singer does that one scream in the song "YAHHHHHH!" or there's a sudden gong sound effect...just something wacky that occurs in the song against the norm.
\nSo you have a few view modes at your disposal. Even more than that, pressing the orange "Analyzation Settings" button will bring up the settings window. Here you can change how quiet your quiet breaks need to be or how long they are in order for them to be identified as quiet breaks.
\nTheres a few settings for every view mode. I'll be tweaking these as I go along to see what works best, and probably change names and defaults.
\n![\"sambaverse_settings\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2010/08/sambaverse_settings.jpg\")
Finally, you can save this data as XML. Actually - I'm not really saving it yet....just posting it to a webpage, and you can save it if you want. Whatever. I know I'm psyched to play around with my new app, tweak some songs, and make some cool games.
\nUnder the Hood
\nI learned a lotta cool stuff making this. The biggest thing of course was how to manipulate and use audio for my bidding in Flash. I used Pixel Bender to get a huge speed increase when loading the sound initially. That was a lot of fun once I figured out how to get it working.
\nAlthough a little annoying at the time, I was using the latest releases of Flex and Swiz. As I've been working on this app for a little while, I had to go through and upgrade from Flex Gumbo to Beta 1 to Beta 2 to RC to Release. Stuff changed every time. Likewise, I had an annoying evening transitioning from Swiz 0.64 to 1.0.
\nBut I'm happy I did this. Flex skinning is awesome. Swiz allowed some nice shortcuts and organization in my application. I even tapped into Flash Catalyst for the initial application design. Catalyst seemed a little limiting after the initial graphics dump. After that for quicky and dirty graphics, I just cracked open Illustrator and exported to FXG which I then copied as text into my MXML skins.
\nfuture improvements
\nI'm going to take a break from this application for a bit and focus on some games. I'm sure I'll tweak as I go along and I find limitations that I need to solve as I'm working on the games. One huge feature I'd like to add when I get back to this is to be able to run a Fast-Fourier Transform on the audio and drop frequencies I don't care about. This way, I can zero in on the drums or the bass guitar and only analyze those. In rock music especially, the frequencies are all over the place at different tempos and can make it a little hard to find a rhythm.
\nI'd also like to make it so you can load in a previously saved XML file to get all your work back for a particular song.
\nLastly I'd like to get a sample game/animation you can preview as you work to visualize the beats better than the Flash lights I have.
\nSo that's Sambaverse! Check it out! (but be nice, it's only an alpha) .
\n![\"sambaverse_main\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2010/08/sambaverse_main.jpg\")
For the 2010 360Flex in Washington DC, I was asked to make a promotional flyer for their after hours music extravaganza.
\nI traced out some photos of the performers to make some line drawings, then superimposed them over the capitol building. Came out pretty cool!
\n![\"360flexapalooza\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2011/08/360flexapalooza.jpg\")
I just had to add a drop down list into my Flex 4 project, but I wanted to be able to multi-select the items, and add some custom logic. It was sort of a pain in the butt figuring out how to do this, so I thought I'd share!
\nSo, first of all, I created a new class that overrode DropDownList. I added this "MultiSelectionDropDownList" to my application. I set the width to 300, set "requireSelection" to false (since we're allowing checkbox selection which the DDList wouldn't know about), and set up a data provider with a few items in my list.
\nAll normal everyday Spark DropDownList stuff so far, right? OK, next, lets create a skin for the list. The only reason I do this is to remove the "maxHeight" for the list. I don't want my list to scroll....because frankly, this isn't the greatest example and will break when my list items are outside the viewable area.
\nNext I'll add an item renderer. The item renderer will be a normal background, and a HGroup on top of it which shows a label and a checkbox (needs an ID for access). Pretty simple stuff so far!
\nNow, I have my multiselection dropdown in my application like so:
\nAt this point we should have a functional drop down showing some items with checkboxes. Of course those checkboxes don't work yet, and we'll need to add some logic.
\nLet's dive into the class we created that overrides the DropDownList - the "MultiSelectionDropDown".
\nThe first thing I want to do is in the constructor - I want to add an event listener for when the list opens:
\n\t\tpublic function MultiSelectionDropDown()\n\t\t{\n\t\t\tsuper();\n\t\t\tthis.addEventListener(DropDownEvent.OPEN, onOpen, false, 0, true);\n\t\t}\n\n\t\tpublic function onOpen(event:DropDownEvent):void {\n\t\t\tactivateAllCheckBoxes();\n\t\t}\n\n\nSo now, whenever someone clicks and opens the list, we'll be "activating" all of our check boxes. What does this mean? Well, we're going to loop through all of our list items, and add some event listeners to these checkboxes.
\n\t\tprotected function activateAllCheckBoxes():void {\n\t\t\tfor (var c:int = 0; c < dataGroup.numElements; c++) {\n\t\t\t\tvar obj:MultiSelectItemRenderer = dataGroup.getElementAt(c) as MultiSelectItemRenderer;\n\n\t\t\t\tobj.checkbox.addEventListener(MouseEvent.MOUSE_DOWN, mouseCheckBox, false, 0, true);\n\t\t\t\tobj.checkbox.addEventListener(MouseEvent.MOUSE_UP, mouseCheckBox, false, 0, true);\n\t\t\t\tobj.checkbox.addEventListener(Event.CHANGE, changeCheckBoxSelection, false, 0, true);\n}\n\n\t\tprotected function mouseCheckBox(event:Event):void {\n\t\t\tevent.stopImmediatePropagation();\n\t\t}\n\nOK, so we've added some event listeners. I'll get to the actual code to change the checkbox selection in a sec - but check out "mouseCheckBox". Here we're grabbing the event, and stopping it's propagation. This means that the buck stops here when you click a check box. The event won't make it's way down to your item, and won't select your item in the list and close the dropdown. Great! Cause, we want to make multiple selections without having to bother with opening the list over and over again.
\nNow, lets focus on the actual clicking of the checkbox:
\n\t\tprotected function changeCheckBoxSelection(event:Event):void {\n\t\t\tcurrentlySelectedCheckBoxes = selectedCheckboxes;\n\n\t\t\t// turn on multi-view mode\n\t\t\tif (event.currentTarget.selected == true ) {\n\t\t\t\tselectedIndex = -1;\n\t\t\t\tlabelDisplay.text = "multiple selections";\n\t\t\t}\n\n\t\t\tdispatchEvent(new Event("selectionChange"));\n\t\t}\n\n\t\tprotected function get selectedCheckboxes():Array {\n\t\t\tvar returnList:Array = new Array();\n\t\t\tfor (var c:int = 0; c < dataGroup.numElements; c++) {\n\t\t\t\tvar obj:MultiSelectItemRenderer = dataGroup.getElementAt(c) as MultiSelectItemRenderer;\n\t\t\t\tif (obj && obj.checkbox.selected) {\n\t\t\t\t\treturnList.push(obj.data);\n\t\t\t\t}\n\t\t\t}\n\t\t\treturn returnList;\n\t\t}\n\nSo here everytime I click a checkbox, if I turned it on (or checked it), I change the label display to indicate that I'm making multiple selections and set the selected item index to -1. This will make the list indicate to the user that nothing is selected, but we're in "multiple selection mode".
\nAdditionally, I'm saving the array of my "currentlySelectedCheckBoxes". I get this array by looping through my items again, and looking at the checkbox testing if it's selected.
\nOnce this is saved, I dispatch a "selectionChange" event. What's this? Well, I made it up, and I'm only using it internally in this class. Who's listening? This guy....
\n[Bindable(event="selectionChange")]\n\tpublic function get selectedViews():Array {\n\t\tvar multiSelect:Array = selectedCheckboxes;\n\t\tif (multiSelect.length > 0) {\n\t\t\treturn multiSelect;\n\t\t} else {\n\t\t\treturn [selectedItem];\n\t\t}\n\t}\n\nThis is a bindable function to provided external access to the array of selected items in the list, whether they are selected through the checkboxes, or just selected in non-multi-selection mode just by clicking on the list item.
\nLet's also improve the experience by using our saved array of "currentlySelectedCheckBoxes" and using that to populate the list when the list opens. You see, when our list closes, our items sorta....disappear. It's sort of annoying, but pretty nice for memory management. Anyway as we go through and add our event listeners when the list opens, we can check if the checkbox is in our saved list, and force it to check itself:
\nprotected function activateAllCheckBoxes():void {\n\tfor (var c:int = 0; c < dataGroup.numElements; c++) {\n\t\tvar obj:MultiSelectItemRenderer = dataGroup.getElementAt(c) as MultiSelectItemRenderer;\n\t\t\tif (obj) {\n\t\t\t\t// no check box here, this playback mode only is a solo view\n\t\t\t\tif (dataProvider.getItemAt(c) == "Select This Item Only" ) {\n\t\t\t\t\tobj.checkbox.visible = false;\n\t\t\t\t}\n\n\t\t\t\t// find and check of previously checked boxes\n\t\t\t\tif ( currentlySelectedCheckBoxes.indexOf(dataProvider.getItemAt(c)) != -1 ) {\n\t\t\t\t\tobj.checkbox.selected = true;\n\t\t\t\t}\n\n\t\t\t\tobj.checkbox.addEventListener(MouseEvent.MOUSE_DOWN, mouseCheckBox, false, 0, true);\n\t\t\t\tobj.checkbox.addEventListener(MouseEvent.MOUSE_UP, mouseCheckBox, false, 0, true);\n\t\t\t\tobj.checkbox.addEventListener(Event.CHANGE, changeCheckBoxSelection, false, 0, true);\n\t\t\t}\n\t\t}\n\t}\n\nBut that's not all! I've added a "solo-selection" click in my dropdown. Clicking this "solo-selection" will clear out all checkmarks and start from scratch - exiting the multiple selection mode. You might notice that I snuck in a line to hide the checkbox if the list item is that "Select this item only" item. That's because it's SOLO selection and it doesn't get a checkbox so we can select it and other items.
\noverride protected function item_mouseDownHandler(event:MouseEvent):void {\n\t// launch solo view mode when clicking "playback only"\n\tif ( event.currentTarget.data == "Select This Item Only" ) {\n\t\tdeselectAllCheckBoxes();\n\t}\n\n\tif (selectedCheckboxes.length == 0) {\n\t\tsuper.item_mouseDownHandler(event);\n\t\tdispatchEvent(new Event("selectionChange"));\n\t} else {\n\t\tcloseDropDown(false);\n\t}\n}\n\nprotected function deselectAllCheckBoxes():void {\n\tcurrentlySelectedCheckBoxes = [];\n\tfor (var c:int = 0; c < dataGroup.numElements; c++) {\n\t\tvar obj:MultiSelectItemRenderer = dataGroup.getElementAt(c) as MultiSelectItemRenderer;\n\t\tif (obj) {\n\t\t\tobj.checkbox.selected = false;\n\t\t}\n\t}\n}\n\nTo accomplish this, I overrode the "item_mouseDownHandler". Inside here, I said, "if you're that solo button, let's deselect all the checkboxes". To deselect, I simply went through each item in the datagroup, accessed it's checkbox and turned off the selected property.
\nI also used this function to determine if we were in multiple selection mode - if so, then close the dropdown list without committing and values. If I wasn't in multiple selection mode, then just do what the list does normally - except dispatch that change event so our custom bindable property updates any outside listeners that our selection has changed (even though it's just a single item now in the array).
\nFinally the very last thing I did....
\nprotected override function dropDownController_closeHandler(event:DropDownEvent):void\n{\n\tif (currentlySelectedCheckBoxes.length > 0) {\n\t\t// if checkboxes are selected prevent the default behavior,\n\t\t// which is to set a selection index\n\t\tevent.preventDefault();\n\t}\n\tsuper.dropDownController_closeHandler(event);\n}\n\nWhen closing the list, normal behavior is to commit a selection. I actually don't know the details of what it was doing - but it was grabbing a selection from somewhere and producing some funky results I didn't want if I had multiple checkboxes selected. It would update the label property and selected item with the last thing I selected in my normal list (even if it was a while ago). It's like the list was remembering something from somewhere and competing with me! Not good.
\nSo, I overrode the close handler. If I had multiple things selected, I stole the event, and called preventDefault(). In the DropDownListBase class, it sees this default is prevented and doesn't try to commit whatever the heck it's trying to commit. It's just lets it be, and closes the list and doesn't change anything.
\nThat's it! You can see the list in action, and right click to view the source here:
\n\nIt's such a tiny thing, but ended up being sort of a pain to figure out how to do - so hopefully I save someone else the pain. Oh and one more thing...I don't know that I'm necessarily advocating this kind of component yet. Might be kinda complicated in terms of useability. I'll find out as I use it in my project and tweak it.
\n", "date_published": "2010-06-24T00:00:00Z" } , { "id": "https://benfarrell.com/blog/2010-05-23-music-visualization-and-papervision/", "url": "https://benfarrell.com/blog/2010-05-23-music-visualization-and-papervision/", "title": "Music Visualization and Papervision", "content_html": "After watching Simon Free's Papervision demo at NCDevCon today, I just had to post an old demo I did when I first got my hands on Papervision a couple of years ago. It's a really cool project (Papervision that is), and I hope to play with it again real soon. In this demo, you drive around a submarine through the sea, and various fish come into your view. But then a James Brown song comes on, and all the fish start dancing to the music. Different fish listen to different frequencies. It's fun, but driving the submarine is very hard, because you can lose it off screen....
\nAnyway - spacebar to make the submarine go, and arrow keys to make it turn (arrow keys are relative to which way the sub is facing)
\nPapervision Beats (with Dancing Fish)
\n(by the way, thanks to Troy and Iris Stratton for getting me started with PV3D and Blender!)
\n", "date_published": "2010-05-23T00:00:00Z" } , { "id": "https://benfarrell.com/blog/2010-05-22-do-you-belieeeeve-in-flash-autotuning/", "url": "https://benfarrell.com/blog/2010-05-22-do-you-belieeeeve-in-flash-autotuning/", "title": "Do You Belieeeeve in Flash Autotuning", "content_html": "No, I didn't accomplish the mythical Flash autotune in time for my "Audio Manipulation in Flash and Flex" presentation for NCDevCon this Sunday. But I don't see why it's not possible in the least
\nI didn't accomplish it, not because of Flash, but because of my lack of digital audio experience.
\nAbout a month ago, I was thinking what effect I could dazzle my attendees with, and ONE thought popped into my head: AUTOTUNE!!! This led me down the rabbit hole of crazy amounts of math, signal processing theory, code optimization, and more. I even started reading a free PDF 600 page book by Stephen Smith http://www.dspguide.com/.
\nI've learned about what Fourier Transforms REALLY are, that they are really useful beyond getting frequency data, how to do low pass filters, and tons of other stuff.
\nI really didn't think it would be this involved. This is like an entire field of expertise. I honestly thought I could steal some algorithm online and be good to go. Nope, I'm swimming with using the FFT, then back with an iFFT, convolving signals together, and all this crazy stuff. It's pretty awesome though, the things you can do and learn from audio signals, and any signal in general. I'll definitely be finishing that 600 page book (probably 3-4 times over so I understand it).
\nAnyway, I was able to accomplish pitch shifting in a couple different ways, and riding a voice on top of another tone (which sounds really close to autotune if you could just get rid of that damn tone!)
\nI didn't even think that one of the things that autotuning did was to detect the frequency of a sung note, and step it up or down to the correct frequency. I thought at the beginning that when T-Pain did his thing, he just sung whatever, and the software would push the voice to whatever the producer wanted.
\nWith Flash, I've seen the Audio Processing Library for Flash detect notes in a sound, and as I've said, I can now pitch shift! In truth, the real Autotune by Antares Audio Technologies is said to use a "phase vocoder", which I'm still not up on my theory enough to know what it is. It's probably a combination of smart pitch-shifting coupled with a flange like effect to go all robot sounding.
\nI finally downloaded 10.1 and got my microphone working - I recorded in from the microphone, and played back via the sound buffer (all through the sample data event). I pitchshifted first by just speeding up the tempo so things got all high pitched and fast. But then I grabbed a PitchShifting class by Stephan M. Bernsee that was ported to Actionscript from C# by Arnaud Gatouillat. Using this was VERY processor intensive. In fact, in debug mode, my entire computer was overheating, and it was a crapshoot whether the sound would actually come out right. And then of course the Windows sound buffer kept doing weird snapping/popping noises every so often until I restarted. However, running as a release build seems to work just fine. I now know why Andre Michelle's AudioTool has a warning against using the debug player.
\nBut all in all, it was a great learning experience. I'm embedding the demo I'll be showing at NCDevCon on Sunday. And of course I'll keep learning to become a DSP master, and someday get my Autotune working (hopefully someone beats me to it, and shares the code).
\nAutotune Attempt - You Need Flash 10.1 and a Microphone
\n", "date_published": "2010-05-22T00:00:00Z" } , { "id": "https://benfarrell.com/blog/2010-05-17-ncdevcon-2/", "url": "https://benfarrell.com/blog/2010-05-17-ncdevcon-2/", "title": "NCDevCon", "content_html": "Just wanted to post a quick update that I'll be speaking at NCDevCon with a hands on Flex 4 session on Saturday with Adrian Pomilio, and then on Sunday with a Flash Audio session. Adrian and I will also be pushing our new group Tri-Me! Check out ncdevcon.com for more info on the conference! It's a great free CF, Flash, Flex, and Web conference in Raleigh
\n![\"Speaking\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2010/05/200x200-Speaking1.png\")
A friend of mine recently tweeted that found that he could create custom events for his project, but he found it rather mysterious - and he didn't really understand what he was doing even though he followed the instructions and was able to make them work in his project.
\nSince I use nothing but custom events in my projects, I thought they'd be a good blog post subject.
\nSo.....custom events. Why do you want them?
\nWell, first lets think about the everyday Flash events. I use flash.events.Event and flash.events.MouseEvent all the time. There are the obvious uses, like when you click on something - you listen for the "click" event type by saying addEventListener("click", onClick).
\nBut, how do you know it's "click"? It could be anything. I hate looking at the docs for every little thing. So, it's much easier to type "MouseEvent." into my code editor and watch it auto suggest stuff for me. You can look through and see MOUSE_DOWN, MOUSE_UP, CLICK, etc, all as options.
\nSo I end up with addEventListener(MouseEvent.CLICK, onClick). It's easy, there's no chance I mistyped it because I didn't type it - my code editor did.
\nBut then you might be moving on to creating your own events. Say you're loading a file. Whether the file loads, fails, the user cancels, you want to know when it's done - so you might dispatchEvent(new Event(Event.Complete)) to let other parts of your project know that the operation is complete regardless of how it completes, just so those other parts know the file loading is done and they can move on with whatever they're doing. Event.COMPLETE is handy for whenever you want to wire up completion events - just like Event.CLOSE is great for when you have stuff that's closing and Event.INIT is great when you have stuff that you're finished initializing.
\nWhat about other types of events? What about events that are specific to your project? For example, say I'm making a game. In the game, I have a rubber ball that the user can interact with. The user can bounce the ball, roll the ball, throw the ball, and spin the ball. The logistics of the user interaction to the ball doesn't matter here, we're just worried about the events that get dispatched.
\nLet's name our events:
\n1. "bounceBall" 2. "rollBall" 3. "throwBall" 4. "spinBall"
\nNow, I could simply just use a normal Flash event. My ball dispatches events in the fashion of "dispatchEvent(new Event("bounceBall"));"
\n...and I listen to my ball bouncing by adding an event listener: "ball.addEventListener("bounceBall", onBounceBall);"
\nThat would work OK, but I'd have to keep looking up what I named the different events, and make sure they match in all parts of my project. God forbid I make a typo - I could be stuck for hours with a "bounceeBall" event not knowing that I accidentally typed an extra "e" in my dispatcher.
\nNo - I like to name my events in static constants, and put them inside the event. That way I can start typing "BallEvent...", and my code editor will autofill the rest.
\nSo the first step is to make a BallEvent by overriding the Flash Event class:
\npackage events\n{\n\timport flash.events.Event;\n\n\tpublic class BallEvent extends Event\n\t{\n\t\tpublic static const BOUNCE_BALL:String = "bounceBall";\n\t\tpublic static const THROW_BALL:String = "throwBall";\n\t\tpublic static const ROLL_BALL:String = "rollBall";\n\t\tpublic static const SPIN_BALL:String = "spinBall";\n\n\t\tpublic function BallEvent(type:String, bubbles:Boolean=false, cancelable:Boolean=false)\n\t\t{\n\t\t\tsuper(type, bubbles, cancelable);\n\t\t}\n\t}\nSo I have a nice little BallEvent class. Now there's no chance of typos. I can dispatchEvent(new BallEvent(BallEvent.ROLL_BALL)) and addEventListener(BallEvent.ROLL_BALL, onRollBall)); and if I DO make a typo, my code compiler will complain and tell me where I went wrong before my SWF ever gets created.
\nOne thing worth mentioning is the "public static const". If you aren't aware of this syntax - a static is a variable that is accessed as a singleton. That is, you don't need to create a new BallEvent object in memory to use it - just say BallEvent.ROLL_BALL, and you'll get the value: "rollBall". Also, a "const" is a constant value - it's something that never changes and you hardcode it ahead of time and don't set it at runtime.
\nSo that's one advantage of custom events - and a good one. Things are readable and typos don't exist. Furthermore, things can be grouped pretty well. My BallEvent class can have all my BallEvents whereas my FrisbeeEvent class can have all my Frisbee events.... it just leads to good organized code.
\nNow that's all well and good - but maybe you said to yourself "Self, it's great I know that my Ball is bouncing, but what time did it bounce, and at what angle, and at what velocity?
\nWell, I could receive the Ball bounce event, and say: var bounceAngle = ball.bounceAngle;
\nBut what good is that? The angle of the bounce is a snapshot of a moment in time. Maybe the angle has changed in the time between when I got the event and when I'm asking the ball what the angle is. All these things can change - ESPECIALLY when you change them yourself. What if you had special code to increase the velocity of the ball when it bounces off something? Well, guess what? You could increase the velocity on the ball bounce event, but then later on in your code, you could try to get the velocity. These order of event things can get confusing. Are you getting the velocity before the you changed it? Or is it after you changed it? It can get downright undependable and muddled.
\nHowever, if you included these values in your event, you can insure that you are grabbing these properties at the moment in time that they happened, whenever you want rather than relying on guesswork.
\nSo let's add a couple properties to the BallEvent. I want to track velocity, time, and angle.
\npublic static const SPIN_BALL:String = "spinBall";
\n public var time:int;\n public var velocity:int;\n public var angle:int;\n\n\t\tpublic function BallEvent(type:String, eventTime:int, eventVelocity:int, eventAngle:int, bubbles:Boolean=false, cancelable:Boolean=false)\n\t\t{\n time = eventTime;\n velocity = eventVelocity;\n angle = eventAngle;\n\t\t\tsuper(type, bubbles, cancelable);\n\t\t}\n\t}\nSo here, I just added the properties I want to the event. When I dispatch the event, I pass them in: dispatchEvent(new BallEvent(BallEvent.ROLL_BALL, 50, 21, 90));
\nAnd then when I get receive the event, I can get all the properties I need!
\npublic function onBallRoll(event:BallEvent):void { var myAngle:int = event.angle; }
\nThat's the basics of the custom event - but there are some best practices that folks like to follow....
\nThe first one is to implement a "clone" method. Let's say that you listened and caught the event, but you really just want to pass it on to another class by dispatching the event from the class that caught it. Well you can call event.clone(), and you'll get a copied event. Unfortunately, those extra properties you added won't be cloned. So let's look at a sample overridden clone method:
\n\tpublic override function clone():Event {\n\t\tvar event:BallEvent = new BallEvent(this.type, this.time, this.velocity, this.angle, this.bubbles, this.cancellable);\n\t\treturn event;\n\t}\nSo here you're actually creating a new BallEvent in this method and returning it when you clone. You're using your public properties that you set when you originally made the event, and just rolling them into this new event. The only thing a little unintuitive here is that the return type is a normal Flash Event. Well.. it needs to be, because you're overriding the Flash Event's clone function (Flash will complain otherwise), but don't worry - when you receive the event it will be cast as the BallEvent with all the properties you need.
\nFor example, in:
\npublic function onBallRoll(event:BallEvent):void
\nThe event, if it originally started off as a BallEvent (even if it got casted to Event through cloning or otherwise), will still come through here as a proper BallEvent.
\nHonestly, sometimes I get lazy and don't implement a clone event. Then several weeks later I need to clone the event and I curse my laziness. But sometimes that never happens. So it's up to you if you think you need it.
\nAnother "pro-tip" is to use getters and setters. Remember the concept of your properties, and how they represent a snapshot in time of the Ball? Well, some people feel that this isn't good enough. Say I'm an evil programmer, intent to sabotage you and your events. I could listen for your event, program a line that says
\nevent.time = 9999;
\n....and then clone and dispatch your event. You'd never catch on to my evil attempts. I've changed a property on the event - and then I passed the event on like nothing happened. It no longer represents a true snapshot in time - more like a doctored snapshot in time.
\nSo, some folks try to stop this. They create getters but no setters. That way if an external class tries to change the property, it will cause your compiler to throw errors, thus saving you from malicious programming.
\nLet's take a peek at a prim and proper finished BallEvent:
\npackage events\n{\n\timport flash.events.Event;\n\n\tpublic class BallEvent extends Event\n\t{\n\t\tpublic static const BOUNCE_BALL:String = "bounceBall";\n\t\tpublic static const THROW_BALL:String = "throwBall";\n\t\tpublic static const ROLL_BALL:String = "rollBall";\n\t\tpublic static const SPIN_BALL:String = "spinBall";\n\n protected var _time:int;\n protected var _velocity:int;\n protected var _angle:int;\n\n public function get time():int {\n return _time;\n }\n\n public function get velocity():int {\n return _velocity;\n }\n\n public function get angle():int {\n return _angle;\n }\n\t\tpublic function BallEvent(type:String, eventTime:int, eventVelocity:int, eventAngle:int, bubbles:Boolean=false, cancelable:Boolean=false)\n\t\t{\n _time = eventTime;\n _velocity = eventVelocity;\n _angle = eventAngle;\n\t\t\tsuper(type, bubbles, cancelable);\n\t\t}\n\n\t public override function clone():Event {\n\t\t var event:BallEvent = new BallEvent(this.type, _time, _velocity, _angle, _bubbles, _cancellable);\n\t\t return event;\n\t }\n\t}\nWell, OK maybe if I was being all proper, I'd throw some ASDoc or some other type of comments in there - but you get the picture.
\nBut my last point here is that I've set all my variables as protected (and I usually put an underscore in front of my protected variables because of a habit I have). The protected variables can't be get or set except by the BallEvent class or anything that extends it (like a RubberBallEvent class or a BaseBallEvent class). So these properties are completely locked in and can't be changed by other classes. But I make them accessible through my getters.
\nThere are other little tricks I sometimes do too. Like in a current project I have a dashboard full of buttons. When I click my dashboard, I dispatch a Dashboard.CLICK event type. But then the actual button clicked is a property called ID, or maybe it's a reference to the actual button.
\nThis saves me a step. Instead of adding an event listener for each and every button in my dashboard, I can simply add one event listener to my dashboard for clicking an event. And then I can have logic in my event handler for deciding what to do based on what event was clicked.
\nEvents are usually pretty simple too, but sometimes you can make them more complex. Like say you could dispatch an event with a time property, but then have several methods to format that time property however you like. Maybe your time is originally a huge number like 22531245665.
\nGetting this time might be useless to display to normal people. Maybe you want to include a formattedTime function in your event. Instead of getting event.time, you could get event.formattedTime and get "April 17, 2010 2:03PM".
\nSo that's how to do custom events! Hope it helps.
\n", "date_published": "2010-04-08T00:00:00Z" } , { "id": "https://benfarrell.com/blog/2010-03-24-using-pixel-bender-to-process-lots-of-data/", "url": "https://benfarrell.com/blog/2010-03-24-using-pixel-bender-to-process-lots-of-data/", "title": "Using Pixel Bender to Process Lots of Data", "content_html": "In my previous post, I talked about using Pixel Bender in the way it was intended. By this, I mean to input an image, change the pixels in some way, and output an image. As you might be aware, this can happen very quickly and be used to create some great run time visual effects in Flash and Flex.
\nWhat piqued MY interest in PxB though, was the fact that the you didn't have to send it an image to process. It could be any ByteArray object. My interest was a little bit more specific - a ByteArray extracted from a flash.media.Sound object. At 44,100 samples per second, with typical MP3 files being 180-200 seconds, we're talking about pushing 10 million samples.
\nMy project involves visualizing the waveform of a song, so I want to process all of these samples at once, and as quickly as I can.
\nSo, lets look at how you would normally apply a PxB shader. The following example is taken from Mike Chamber's blog:
\nimport flash.filters.*;\nimport flash.utils.ByteArray;\n\n//the file that contains the binary bytes of the PixelBender filter\n[Embed("testfilter.pbj", mimeType="application/octet-stream")]\nprivate var TestFilter:Class;\n\nprivate function onApplicationComplete():void {\n//Pass the loaded filter to the Shader as a ByteArray\nvar shader:Shader = new Shader(new TestFilter() as ByteArray);\nshader.data.amount.value = [100];\nvar filter:ShaderFilter = new ShaderFilter(shader);\n\n//add the filter to the image\nim.filters = [filter];\n}\nSo, Mike's got the right idea here. He creates a shader, makes a shader filter out of it, and then applies it as a filter to the image.
\nWe're going to be doing something slightly different. Instead of a ShaderFilter, we'll be creating a ShaderJob. And we'll also need to tell our ShaderJob certain things that our ShaderFilter knew automatically. Specifically, we're talking about the height, width, and input source of the image. After setting these properties you add an event listener to the job, and tell it to start.
\nvar output:ByteArray = new ByteArray();\ndata.position = 0;\n\nvar channels:int = 4;\nvar width:int = 2000;\nvar height:int = data.length / width / (channels*channels);\n\nshader.data.src.width = width;\nshader.data.src.height = height;\nshader.data.src.input = data;\n\nvar shaderJob:ShaderJob = new ShaderJob(shader, output, width, height);\nshaderJob.addEventListener(Event.COMPLETE, onComplete, false, 0, true);\nshaderJob.start();\nOK, I'll back up...it's a lot to start with. Let's talk about the input data first - our byte array. Because I'm talking specifically here about a byte array extracted from sound, I know that my byte array is a long list of 4 byte floating point numbers. This particular list of numbers alternate from left channel to right channel, but that's neither here nor there until we get into the Pixel Bender kernel.
\nSo, now we're talking the language of Pixel Bender - a byte array of 4 byte floats. Now, we're typically we're dealing with 4 of these 4 byte floats (formerly known as red, green, blue, and alpha). I believe that PxB is SUPPOSED to accept image1, image2, image3, or image4 type inputs, however I've read there are bugs associated with doing anything less than image3 as the input. Even if we wanted to take advantage of an image2 - think about it. PxB is designed to process a pixel as fast as it can, Why not take advantage and create the largest pixel we can? Little decisions like this multiply in big ways when you start talking about your entire data set, and you could be talking about saving hundreds of milliseconds for these little decisions.
\nSo, each pixel is 16 bytes. In my sample code, I'm calling out 4 "channels". This is an important variable when thinking about our height and width.
\nLet's talk a little about height and width now. As you know, an image has a height and width. But how does this apply to a one dimensional list of items for PxB?
\nIt all comes down to speed and optimization. You could certainly give your data input a height of 1 and a width of 8192 (the height and width limit) and treat your data as a one-dimensional list. It's certainly easier to picture that way, but then you wouldn't be taking advantage of PxB's built in speed and optimization.
\nPixel Bender is multi-threaded, multi-core, and GPU enabled. While that last part doesn't hold true for Flash, the first two parts are important. Not only can PxB itself operate on a different thread and CPU core, so can each row of data. So, PxB can be processing many rows of pixels all at the same time. If you only do one row per job (a height of 1), you're throwing away most of the speed benefits of Pixel Bender! The data itself will wrap around to the next row, just like a text field wraps text to the next row.
\nHow does you know what width and height to give your data? Well, with a possible width and height of 8,192, you're giving yourself a possible 67 million pixels to work with in one job. Each one of these pixels holds 4 channels. Which means, in one job you can process 268 million floating point numbers.
\n![\"pxb\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2010/03/pxb1.jpg\")
Earlier I had talked about each MP3 file being around 10 million samples. I'm personally conducting my own speed tests, but I've found that processing 60 seconds of an MP3 file at a time is a little better overall for speed. So we're only talking about 2.5 million samples that I would personally process at a time. So, now the question becomes: How can we reasonably distribute our data as an image. Well, if we max out our width, we may not be producing many rows. As a result, speed may be reduced because we're not utilizing multiple threads as efficiently as we could. Based on my input data, I've found that a width of 2000 works well for me based on the specifics of my application. Your mileage may vary, so feel free to try different things. Just remember, you'll want to take advantage of many rows for PxB optimization.
\nAnd then of course, to get the height, you can work that out by calculating this from your width and the overall length of your byte array.
\nAfter all has been said and done, you start the job, set the event listener and wait for the job to complete.
\nNext up, I'll be posting some speed tests!
\n", "date_published": "2010-03-24T00:00:00Z" } , { "id": "https://benfarrell.com/blog/2010-03-24-pixel-bender-speed-tests/", "url": "https://benfarrell.com/blog/2010-03-24-pixel-bender-speed-tests/", "title": "Pixel Bender Speed Tests", "content_html": "This is my last in a series of blog posts on Pixel Bender. I'll probably be doing one more PxB post when I upgrade to Swiz 1.0 and we're talking about PxB processors with Swiz, but until then....
\nThis post will be focusing mostly on speed in Pixel Bender and doing similar operations in normal Actionscript.
\nPixel Bender is good on speed, pixel by pixel, but there are some operations that make PxB a little less efficient. In my tests - these efficiency deficiencies are made worse with specific operations.
\nThe worst is when you sample the nearest pixel. In PxB, sampling one pixel is fine. But the more pixels you sample as you evaluate one pixel, the worse this problem is exasperated.
\nWhy sample more than one pixel? Well, a blur effect is a great example. When you blur, you're probably averaging the center pixel and the 8 surrounding pixels. Or downsampling an image...you'd be sampling a whole mess of surrounding pixels and reducing to 1 pixel. In my audio visualization application, I'm not reasonably going to view 2.5 million pixels. No, I'm going to reduce down to something that can be displayed - maybe a few thousand, depending on how far I zoom.
\nIn any of these scenarios, I'm sampling more than one pixel. The more pixels I sample, the worse the performance gets. In my experiments, it can get to the point where PxB is no better than just doing things in Actionscript. In these cases, at least Pixel Bender has the whole multi-threaded feature going for it, so your UI performance won't suffer.
\nAnother horrible aspect of sampling more than one pixel is that PxB for Flash doesn't support loops and functions. So, if your kernel needs to sample 100 neighboring pixels, you have to write the 100 lines of code to go with it.
\nFor audio processing, sampling surrounding pixels is the whole point. The key is finding the right amount of pixels to sample where you get a good average, but still have your Pixel Bender shader running lean and mean.
\nWith that said here's a sample application which demos performing math operations in Pixel Bender vs performing the same operation in Actionscript 3.
\nPixel Bender Speed Test Application
\nKeep in mind that we keep all the loading of the MP3, and the extraction of the byte array at the very start of the application. These actually take a fairly long time (almost a full second to extract the audio), but are a necessary evil whether you use Pixel Bender or you don't use Pixel Bender. Lets look at the results:
\nSimple Copy Operation
\nTake data, and return the same data (but make sure to copy each and every value to a byte array one at a time)
\nFlash AS3: 1.715 seconds
\nPxB: 0.475 seconds
\nPower of 2 Operation (Square a Value)
\nTake each value, square it, and copy into a new byte array
\nFlash AS3: 2.81 seconds
\nPxB: 0.825 seconds
\nMultiply by 12.54 (a random number I picked for this test case)
\nMultiply each number by a value and copy to a new byte array
\nFlash AS3: 1.747 seconds
\nPxB: 0.495 seconds
\nDo a Complex Math Operation
\nTake each value, multiply by 12.54, divide by 100, take the sin of that, and then take the square root of the result.
\nFlash As3: 3.53 seconds
\nPxB: 0.713 seconds
\nAverage 10, 20, 50, and 100 values
\nThis is where Pixel Bender should start getting less effective, as we are sampling more and more surrounding pixels.
\nAverage 10 - Flash AS3: 3.727 seconds
\nAverage 10 - PxB: 0.878 seconds
\nAverage 20 - Flash AS3: 3.954 seconds
\nAverage 20 - PxB: 1.153 seconds
\nAverage 50 - Flash AS3: 3.613 seconds
\nAverage 50 - PxB: 2.047 seconds
\nAverage 100 - Flash AS3: 3.636 seconds
\nAverage 100 - PxB: 3.469 seconds
\nAs you can see, Pixel Bender does VERY well until you start sampling more and more neighbors. As we reach 100 neighboring pixels sampled, there really is no speed advantage to Pixel Bender. Fortunately though, we're still talking about an operation on a different thread that won't slow down the UI. So even if it does take 3 and a half seconds, its 3 and a half seconds of time that our UI is as snappy as ever.
\nRemember before when I talked about extracting your audio? Well, you don't have to grab 60 seconds all at once and lock up your UI for a full second. You could grab less, and process the audio in increments. With PxB, you can just keep running shader job after shader job. You'll have to determine what's best for your UI vs how fast you want the operation to complete.
\nAnother thing that I've hidden (though I didn't mean to) in this demo application is the time it takes to create the Shader. I've found that it takes around one second to create a Shader. This means that before you utilize a Pixel Bender kernel, you must take 1 second of unresponsive UI time to have a shader be created. Creating a ShaderJob, on the other hand takes about a millisecond. This means that you could spend 1 additional second at the beginning of your application to initialize the shader. That's one second of time that you didn't have to spend running your AS3 only code. However, the more ShaderJobs you have to split up the work, the more this 1 second becomes a moot point.
\nAn additional note of interest is the shaderJob.start method. By default, the parameter you pass in here is "false". This boolean indicates whether to "waitForCompletion". If you don't wait for completion, then you are using this event asynchronously - meaning you assign an event complete listener and wait for the operation to complete while your code does other things. If you specify the "waitForCompletion" flag as true, then all Flash operations will halt until your shader job finishes, and you don't have to set an event complete listener because the next line of your AS3 code will just execute after the job completes. I've read that this can give you a slight speed increase, but I haven't had this experience. Plus it will lock up your Flash code execution until it completes, meaning your UI will be unresponsive.
\nAnd that completes my series of posts on Pixel Bender for now. Kevin Goldsmith at Adobe has been doing some pretty cool stuff as well. He does audio processing live, or rather each time Flash reaches into the audio buffer to get the next sound samples. So he actually uses Pixel Bender to alter audio AS YOU PLAY IT. So that's very cool, check it out here.
\nThere is much more to this Pixel Bender stuff. Find it online, and especially read up on the Pixel Bender Twiiter account. For now though, I'm done. Happy Bending!
\n", "date_published": "2010-03-24T00:00:00Z" } , { "id": "https://benfarrell.com/blog/2010-03-24-pixel-bender-roundup/", "url": "https://benfarrell.com/blog/2010-03-24-pixel-bender-roundup/", "title": "Pixel Bender Roundup", "content_html": "Hey everyone,
\nI just gave a Pixel Bender presentation at our local Adobe User Group yesterday. Here's a roundup of the blog posts I wrote as I was collecting my thoughts:
\nFirst I wrote about the problems I wanted to solve, and why Pixel Bender seemed like the right choice:
\nhttp://www.blastanova.com/blog/2010/03/19/pixel-bending-for-speed-in-flash-and-flex/
\nNext, an overview and intro of Pixel Bender:
\nhttp://www.blastanova.com/blog/2010/03/24/intro-to-pixel-bender/
\nAnd then, a post about using Pixel Bender for data instead of images:
\nhttp://www.blastanova.com/blog/2010/03/24/using-pixel-bender-to-process-lots-of-data/
\nFinally, some speed tests along with a sample application I wrote to run the speed tests:
\nhttp://www.blastanova.com/blog/2010/03/24/pixel-bender-speed-tests/
\nHappy Pixel Bending!
\n", "date_published": "2010-03-24T00:00:00Z" } , { "id": "https://benfarrell.com/blog/2010-03-24-intro-to-pixel-bender/", "url": "https://benfarrell.com/blog/2010-03-24-intro-to-pixel-bender/", "title": "Intro to Pixel Bender", "content_html": "So this is my intro to using Pixel Bender - if you don't know what it is or why to use it check out the documentation or my first post.
\nActually the documentation is a good place to start. Go to http://www.adobe.com/devnet/pixelbender/ before you begin. Go on....I'll wait...
\nOnce there you'll want to download the Pixel Bender Toolkit. It's a simple and light program - not subject to the lengthy installs of other Adobe software. Don't bother downloading the PDF documentation, there are links in the help menu to these documents once you run the toolkit.
\nThe best part of the documentation - which actually turns into the most depressing part once you learn it, is that you ignore around half of it if you're developing Flash shaders. Most of the more advanced functionality only applies to Photoshop and After Effects shaders.
\nSo crack open the toolkit! What to do now? Well first, go to the file menu and choose "new kernel". Kernels are basically the "programs" you're creating that compile to shaders. A new kernel will look like this:
\nkernel NewFilter\n< namespace : "Your Namespace"; vendor : "Your Vendor"; version : 1; description : "your description"; >\n{\n input image4 src;\n output pixel4 dst;\n\n void\n evaluatePixel()\n {\n dst = sampleNearest(src,outCoord());\n }\n}\nDon't worry about that top part - it's just noting the author of the script.
\nSo the first thing to worry about are the two variables at the top. There's "image4 src" and "pixel4 dst". You might guess that you're defining a source image and a destination image. But what's up with the funny syntax?
\nWell, first of all, Pixel Bender is one of those languages where the data type is in front of the variable. So you have a variable "src" of type image4, and a variable "dst" of type pixel4. Image and pixel datatypes might make sense, but the "4" is what threw me off at first, but don't worry it makes sense.
\nPxB mainly deals in floating point numbers. And no automatic type conversion! Doing float var = 2 is no good, but doing float var = 2.0 is OK. There are basically 4 types of floating point numbers: float, float2, float3, and float4. A float4 is basically an array of 4 floating point numbers. Example: float4 myfloat = float4(2.0, 2.0, 2.0, 2.0);
\nIt only goes up to 4. Once you realize that the main point of PxB is to manipulate pixels, you being to see why. Red + Blue + Green + Alpha = 4 channels and an array of 4 floating point numbers.
\nI've found that I can use pixels and floats interchangeably (maybe I'm wrong). Images are reserved for an entire image comprised of pixel 4's/float 4's. Pixel Bender also supports integers and booleans (each with 4 or less values).
\nOK that was my rant on variables. Lets move onto "evaluatePixels". This is THE method that everything PxB revolves around. In fact, in Flash, you can't even create other methods in your kernel (PS and AE allow this though).
\nEvery PxB kernel is designed to do one thing and one thing only. Take in a source image, go pixel by pixel, and create an output image pixel by pixel from the "evaluatePixels" method.
\nThat's easy enough to understand - but what about that wonky syntax they start you off with?
\ndst = sampleNearest(src,outCoord());
\nSo, lets work from the inner to the outer. Starting with outCoord(). This method gets the current coordinates that pixel bender is analyzing at that moment. Hint: it's a float2, containing both X and Y values.
\nThat was the easy part - the hard part is "sampleNearest()". For us Flash folks, this introduces you to the wholly confusing notion of sampling on half pixels and pixel ratios and other such nonsense. But then you realize that its Flash, and all pixels are square, and sampling a pixel samples the entirety of the pixel. At this point you realize that sampleNearest is just how PxB works - but it's entirely unnecessary for Flash.
\nSo in other image editing software (and apparently this holds true especially for video), pixels don't have to be square. Pixels can have a different height and width, giving them an aspect ratio, which you can actually check for in PxB. ![\"pxbworldcoordinate\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2010/03/pxbworldcoordinate.jpg\")
But then there's PxB...it will scan each pixel in the image as IF THEY WERE square. So you end up with coordinates that could be x:4.56, y:1.567. When you do "sampleNearest", you're sampling the nearest pixel to these fractional values to end up with nice locked-in PxB world coordinates like x:4, y:2. You can also call "sampleLinear" which takes the average of the surrounding pixels when you ask for something on a half pixel.
\nBetcha feel smart now, don't you? Well forget everything you just learned. If you are doing things in Flash, all pixels are square, and all pixels match to the PxB world coordinate system perfectly. So "sampleNearest" is just something you have to do to get the red, green, blue, and alpha values of the pixel.
\nSo - in the end...you're just taking the pixel you've come to, evaluating the 4 channels, and dumping those right back into the destination pixels. In other words, you're doing nothing.
\nAt this point though, it becomes easy to start manipulating an image. Go to the file menu again, and load an image. PxB has some sample ones to use, like this one:
\n![\"Untitled-1\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2010/03/Untitled-1.jpg\")
Now change dst = sampleNearest(src,outCoord()); to:
\ndst = sampleNearest(src,outCoord()) * float4(0.25, 1.0, 1.0, 1.0);
\nNow click "run"
\n![\"reducered\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2010/03/reducered.jpg\")
Congratulations! You just went into every pixel and turned the red down to 25%.
\nHow bout a weird cross-hatch type effect?
\ndst = sampleNearest(src,outCoord()) * float4(sin(outCoord()[0] * 4.0), cos(outCoord()[1] * 4.0), sin(outCoord()[0] * 4.0), 1.0);
\n![\"crosshatch\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2010/03/crosshatch.jpg\")
Play around, try different things. If you break anything, Pixel Bender will give you red error messages of varying usefulness on the right side.
\nThe hardest thing to get used to is always typing numbers with decimals and usually performing operations not with one set of numbers but with a set of 4. If you run into any trouble, keep asking yourself these two questions:
\nSo that's the basics of PxB! You can manipulate surrounding pixels if you like by performing operations on surrounding pixels. Just add or subtract X and/or Y to your outCoord(), and sample that pixel. Combine and average surrounding pixels to get a blur effect for example.
\nHere's an example of taking a big image, and downsampling the image to a tiny corner in the upper left of the destination:
\n float4 colorAccumulator = float4(0.0,0.0,0.0,0.0);\n float4 avg;\n colorAccumulator += sampleNearest(src, outCoord() * float2(9.0, 9.0) + float2(-1.0, -1.0));\n colorAccumulator += sampleNearest(src, outCoord() * float2(9.0, 9.0) + float2(0.0, -1.0));\n colorAccumulator += sampleNearest(src, outCoord() * float2(9.0, 9.0) + float2(1.0, -1.0));\n colorAccumulator += sampleNearest(src, outCoord() * float2(9.0, 9.0) + float2(-1.0, 0.0));\n colorAccumulator += sampleNearest(src, outCoord() * float2(9.0, 9.0));\n colorAccumulator += sampleNearest(src, outCoord() * float2(9.0, 9.0) + float2(1.0, 0.0));\n colorAccumulator += sampleNearest(src, outCoord() * float2(9.0, 9.0) + float2(-1.0, 1.0));\n colorAccumulator += sampleNearest(src, outCoord() * float2(9.0, 9.0) + float2(0.0, 1.0));\n colorAccumulator += sampleNearest(src, outCoord() * float2(9.0, 9.0) + float2(1.0, 1.0));\n\n dst = colorAccumulator/9.0;\nMy next post will be about taking Pixel Bender and using it for non-image data processing. Stay tuned!
\n", "date_published": "2010-03-24T00:00:00Z" } , { "id": "https://benfarrell.com/blog/2010-03-19-pixel-bending-for-speed-in-flash-and-flex/", "url": "https://benfarrell.com/blog/2010-03-19-pixel-bending-for-speed-in-flash-and-flex/", "title": "Pixel Bending for Speed in Flash and Flex", "content_html": "Lately, as I've been progressing with a Flex 4 based application which is largely an audio visualizer, I've felt the pain of slow response times from my UI.
\nThe slow response is due to the fact that I'm loading a 3 minute MP3, extracting the audio to a byte array, and then processing the data in that byte array. Consider a 180 second song with 44,100 samples per second. That's 7.9 million numbers to process as fast as I can.
\nThis has caused me to do some complicated things. The best of my efforts entailed processing the audio in segments on every enterframe handler. I'd tell flash to process a reasonable amount of data, stop, and then process more the next round until everything was finished. If I remember correctly, this took around 20-25 seconds or so, and STILL made my UI very sluggish as I was processing the data. And of course this was in my development environment - in real life, a user would be waiting even more time while an MP3 loads.
\nSo, I had two basic problems. The first was that data takes too long to process. I can hide this potentially with a good user experience - after all people expect that loading a large file can take some time. But this leads to my second problem - Actionscript runs on one thread. Only one thing can happen at a time. If I'm processing audio, I can't be updating/refreshing my UI. And the whole application becomes unresponsive. The more responsive I make my UI, the less data I process in a frame - but this just makes the whole darn thing take longer to process. Worse yet, if I process the whole thing in one go, Flash can timeout from inactivity at 15 seconds.
\nBasically it boils down to running it all at once, and potentially doing this:
\n![\"error1502\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2010/03/error1502.jpg\")
or this....
\n![\"1476guy_asleep_computer\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2010/03/1476guy_asleep_computer.jpg\")
Fortunately, there are two such Flash technologies that had the promise of helping me out. First there's Alchemy. Alchemy is an Adobe research project that compiles C++ code to a Flash SWC. Supposedly it can run code up to 10x faster than code compiled with Actionscript.
\nBut, I haven't touched C++ in a while, and still had the problem of an unresponsive UI for the time it takes to process the audio (whatever that time is). It was tempting, but I had my sites set on trying Pixel Bender.
\nPixel Bender is a cross-product Adobe technology. It runs in Flash, Photoshop, and After Effects. Basically it allows developers to write their own image filters. In Flash, you can apply this image filter to images, animations, video, components.....and well anything that displays on the stage.
\nThe best part? It can run in different threads, on different CPU cores, and on your GPU. Well, actually, strike that last part....you can't run it in Flash on your GPU, but Photoshop and After Effects are cool.
\nThis means that you can run a Pixel Bender shader on an image, and your UI doesn't slow down. Maybe you wrote something insanely complicated, well...then your PxB shader will suffer performance, but your UI won't!
\nIt doesn't stop there - you don't even have to have an image as your...erm...source image. Yes, PxB assumes red, green, blue, and alpha channels. But you can easily lie to PxB, and have it assume that your custom data is RGBA data.
\nThis brings me full circle back to my problem. I have something that supposedly processes data quickly, and on a different thread. Hooray! In fact, there's a few projects going on that use PxB for audio processing already.
\nCome to the March RDAUG meeting on Tuesday to find out more! I'll be presenting, and quickly talking about what I covered here, but also diving into the nitty gritty. You know....how to actually use this stuff in your work.
\nI'll also be following up this blog post with a second part next week covering what I went over in my presentation.
\n", "date_published": "2010-03-19T00:00:00Z" } , { "id": "https://benfarrell.com/blog/2010-02-09-vacation-from-flash/", "url": "https://benfarrell.com/blog/2010-02-09-vacation-from-flash/", "title": "Vacation from Flash", "content_html": "I just came back from a long weekend trip to Orlando, Florida. Just prior to that I had been working to launch this website - complete with a jQuery driven photo album, Joomla based CMS, and WordPress based blog.
\nThis past weekend, waiting in the long Disney World lines, I had plenty of time to read the tweets coming in as the Adobe development teams and the Flash developers I know responded to Flash bashing and HTML 5 superiority in the wake of the Apple iPad announcement and the impending release of Flash Player 10.1, which Adobe is pushing for mobile devices.
\nI'm not going to add anything to this argument, but my current opinion stands that everything has it's time and place. People use Flash horribly just like they'll use Javascript and HTML 5 horribly, and likewise each will produce some great things.
\nWhile most folks are comparing the end result when you create with the different technologies, what was eye opening in creating my website is the comparison in the development process between Flash, Flex, HTML, CSS, and Javascript.
\nJavascript and Actionscript are the most obvious comparisons. Both have pretty much the same syntaxes. Both are ECMA based, however Javascript has a few less features, that to me....can slow development and can lead to poorly written and badly maintained code. Please note that I said CAN slow and CAN lead to!
\nLet's go back in time a little. Flash 5 and 6 were great at the time, but didn't encourage any good programming standards. Basically you'd write scripts on each frame of your animation. You could also (and many people did) go into each object and animation and write scripts on those as well. In the end, you could have a huge unorganized mess which was impossible to comprehend or troubleshoot.
\nIf Flash was your first development environment, over time, you'd start to see that you've created big unorganized messes and start to think about how to better maintain your projects. However, there was absolutely nothing in Flash that encouraged this behavior.
\nFlash 6 and Actionscript 2 started rolling out with new enhancements to the ECMA standards. This included the ability to organize your code in classes, and really rounded out an object oriented approach to development. You still weren't forced to acknowledge OO practices in your work, but the option was definitely there and some of the more experienced programmers could utilize them.
\nWith Flash 9 and Actionscript 3, developers who tried to do anything substantial were forced to start acknowledging OO practices. This approach led to Adobe facing some complaints from the casual Flash developer, but also led to the larger community adopting some good (or better) practices around object oriented development.
\nI think this very much relates to the state of Javascript today. Someone coming from a more robust language like Java or C++ could pick up both Flash 5 or Javascript today. They could absolutely introduce a code structure to mimic an organized object-oriented approach. However, the reason they can do this is because they have the experience to know how to better organize code, and they know right off the bat what features are missing, and can concoct ways around them.
\nContrast this to someone just learning Javascript or Flash 5 as their first development language. They just wouldn't know any better than to create lots of inline code, not realizing that classes or methods even exist. As this person gets more and more experience under their belts, they would see how their approach lacks maintainability and strive for something better. In the interim however, this leads to horrible, mangled, and unmaintainable scripts.
\nActionscript has increased by leaps and bounds over the years, but Javascript still has not. Great code can and will be developed in each environment - but I just don't think that Javascript encourages any good practices.
\nKeep in mind that I'm well aware that the exact same thing can be said about Java/C++ versus Actionscript with Actionscript on the low end of the totem pole here.
\nThe other thing to point out is variable typing. Having to call a variable a string, number, etc when first declaring it seemed like a nuisance, but turned out to speed up development quite a bit. This is an important thing that Javascript lacks.
\nSomeone, a year back, turned me onto Aptana for use when developing with HTML/CSS/Javascript. Aptana is fabulous. I get to use my experience with Eclipse and Flex/Flash, and just switch to something that looks exactly the same to develop my non-Flash web pages with. Code hinting is huge for me. I don't constantly have to look up syntax, and can start typing and have Aptana suggest what to do next. This is incredibly helpful for CSS, since I don't remember all the properties and need constant help.
\nWith Javascript, on the other hand, code hinting is hindered a great deal. If I don't declare a variable type, Aptana can't suggest what properties and methods I can use on an object very accurately. It actually does a great job guessing (especially with jQuery), but it can only go so far if it doesn't know what kinds of objects you're working with.
\nI suppose you could call CSS a win on the non-Flash side. CSS, to me, is simply a joy to work with given the right tools. Aptana, as I said before, does well with code-hinting for CSS. As a Flex developer who knew enough CSS to style text in HTML and style my Flex components, really getting into CSS was a little daunting. It's a lot more powerful and robust than I would have thought. Flex doesn't give you the ability to control layout and positioning with CSS, nor does Flex 3 allow inheritance and chaining styles together (Flex 4 does though!). It can also be a little confusing in Flex to know which properties to set for what you want to do.
\nAll in all, I think I enjoy style and layout in HTML/CSS more. I'm still a newbie, but it's my shiny new toy. Unfortunately, my shiny new toy is very tarnished by having to maintain compatibility in Internet Explorer. I recently tried and failed to use a glow effect for some text that looked great in Firefox and Chrome, but looked awful because of IE's clear type. And then of course, there's no real practical way to use vector graphics (IE doesn't natively support them).
\nFor pure programming power, I'll choose Flex and Flash any day of the week.
\nI look forward to getting more experience with Javascript and CSS. It'll definitely help when I'm evaluating a project. I'm sure I lean too far on the Flash side right now, and I look forward to being able to better approach a project from any perspective.
\n", "date_published": "2010-02-09T00:00:00Z" } , { "id": "https://benfarrell.com/blog/2010-02-03-retiring-yellow5labs-com-launching-blastanova-com/", "url": "https://benfarrell.com/blog/2010-02-03-retiring-yellow5labs-com-launching-blastanova-com/", "title": "Retiring yellow5labs.com, launching blastanova.com", "content_html": "Oh so long ago (2001 maybe?), I started yellow5labs.com. I had been recently laid off, and wanted to set myself up with an online identity to get contract work and look all flashy so people would hire me.
\nThe name yellow5labs was inspired from Yellow #5 food dye. You know, that stuff...that you put into twinkies, and doritos, and all sorts of good stuff to give something plain that extra pizazz. I was kinda thinking I would sell it as I was bringing that pizazz to my interactive work. I still think it's a pretty cool name, except it's horribly impractical.
\nEvery time I have to give my email address, I have to say "yellow.....five.....no, the number five, don't spell it out, labs.com.....no, I don't have 5 Golden Labrador retrievers.....no...FIVE....don't spell it out..."
\nIt got a little tiresome. The site is still up, but I'll probably just let the domain lapse this summer.
\nAnyway, in with Blastanova. If you hear it, I think you'll know how it's spelled, and that should save me a lot of grief. Why blastanova though? Well, lately I'm both fascinated with the idea of music based video games, and sad that I don't like music as much as I used to. I think a lot of it has to do with the fact that I'm not forced to use the CD player in my car anymore, and I have an unlimited amount of music. So songs are almost disposable now, and no real connections are made with the artists, because you aren't listening over an over again to the same disc.
\nI think games rhythm games (like Rock band most recently), really get you listening to the songs over and over again, and really establishing a connection between you and the music.
\nSo - "blast" as in blasting stuff in video games.....and "blastanova" as in bossanova, a music that literally translates to "new beat" and derived from Samba which itself is known for it's rhythm. I dig.
\nIn the short term, I'll be blogging on new media, developing for the web, Adobe's Flash & Flex, and bands I'm listening to.
\nIn the long term, I'm developing an application to scan songs and their rhythm and structure, and soon thereafter, I'll be applying the data to games I develop.
\nSo that's the plan.
\n", "date_published": "2010-02-03T00:00:00Z" } , { "id": "https://benfarrell.com/blog/2009-07-15-composing-music-with-the-flash-bytearray/", "url": "https://benfarrell.com/blog/2009-07-15-composing-music-with-the-flash-bytearray/", "title": "Composing Music with the Flash ByteArray", "content_html": "Demo here, view source for code
\n(view source for code, and who knows what bugs there are)
\nI'm getting into unfamiliar territory these days with exploring sound in Flash.
\nFlash 10 gives you the ability to take a sound object, and extract the entire thing into the raw data....a byte array. How do you interpret this data? How can you process it into something meaningful? Well I don't know.
\nWhat I do know is that to get a firmer grasp on how I can utilize this, I need to understand what the sound object is at a basic level. Probably the worst way to understand it is to take an entire song (mp3) file and let my eyes glaze over at the stream of numbers coming from the sound.extract feature.
\nNo - the best way, I thought, would be to compose my own notes and chords, and work up from there. To do this, I needed to understand what a note is, and what a sound is at a basic level.
\nMy first thought was to picture sound as the nifty looking sound visualization that comes with music players these days...you know, those dancing bars:
\n![\"equalizer\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2009/07/eq1_inline.jpg\")
Turns out that this is the worst way to imagine sound for this purpose. My first question was...."OK, how do I get the value of the low frequency?", "how about something in the midrange?", "what about the highend?".
\nYes you can imagine sound this way, but only after processing your sound data with a Fourier Transform. If you know how to do this, stop reading this right now, cause you're way smarter than I am at this point in my sound exploration.
\nThe BEST way to imagine sound is to picture a sine wave:
\n![\"sine](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2009/07/sinewave.gif\")
If you look at how tall the wave is, that is how loud the sound is, or the amplitude. How close the peaks and valleys are in this picture is the frequency.
\nFrequency, at this very basic level can't be thought of as high pitched, low pitched, etc - it's only how far apart these peaks and valleys are. Now think of this as a side-scrolling wave that goes on forever. If you were to scroll at a constant speed, each peak would hit that vertical center line at a constant rate. This rate would be the frequency. If it goes faster, the frequency is higher - slower, the frequency is lower. And, of course, this directly relates to how you hear it. The more peaks and valleys at a given time, the higher the tone you hear.
\nNow think about how this relates to our Flash sound object's byte array. We could actually draw a sine wave with numbers. This is nothing new to programmers. Folks use trigonometry all the time. But, I personally, never thought to use it for sound.
\nTry this AS3 code:
\nfor (var i:int=0; i < 100; i++ ) { var sample:Number; sample = Math.sin(i * 2 * Math.PI) * 50; }
\nThis will basically plot a picture like the one above - a sinewave.
\nIf we made this into a sound, it almost, but not quite be a tone. However, if we pop over to google, we can actually look up the frequency, of say.....a middle C note, or a middle A.
\nhttp://www.phy.mtu.edu/~suits/notefreqs.html
\nThere's one more piece to this puzzle though, and that's sampling rate. A digital audio file could have all the sinewaves, peaks and valleys in the world, but if your audio playback is super slow, it's gonna sound like garbage. That's why we tell our audio software to read our sound at a certain speed.
\nIf you've heard the term 44.1 kbps when folks talk about sample rate - you can look back at our little for loop that drew the sine wave, and realize that you need more than the 100 points of data as we drew, you need 44.1k, or 44,100 PER SECOND.
\nSo let's rewrite that code:
\nfor (var i:int=0; i < durationinseconds * 44100; i++ ) { var sample:Number; sample = Math.sin((i) * 2*Math.PI/44100 * 440) * volume; }
\nOK! So now, we're creating a one second middle A (4th octave). Our sample rate in Flash is always going to be 44100 if coming from a flash sound object (though that's not true for any audio file). We learned from our handy/note frequency chart when looking on google that 440hz is a middle A.
\nAs a side note, lets say we want a different octave. To go lower, half the frequency to get 220. To go lower, half that. Higher? Double it.
\nLet's think about the voice now. The nice sine wave will give a nice, round tone. For a dirtier tone that sounds like it has sharp edges....well our sine wave needs to have sharp edges - which is actually a square wave. Our nice round peaks and valleys would be just straight corners.
\nTo change the tone in the code, try this:
\nsample = Math.sin((i) * 44100 * 2 * Math.PI * frequency) > 0 ? volume : -volume;
\nAs for more voices, well, I haven't tried it, but a real world instrument would have a hard strike and then some falloff. So our nice sine wave would be less round at the start of each peak, but comes back down way slower.
\nHere's my final code (and keep in mind that I'm writing the sample 2 times, one for the left channel and one for the right):
\nreturnBytes:ByteArray = new ByteArray(); for (var i:int=0; i < _duration * 44100; i++ ) { var sample:Number; if (_voice==VOICE_SQUARE) { sample = Math.sin((i) * 44100 * 2 * Math.PI * this.frequency) > 0 ? _amplitude : -_amplitude; } else { sample = Math.sin((i) * 44100 * 2 * Math.PI * this.frequency) * _amplitude; } returnBytes.writeFloat(sample); returnBytes.writeFloat(sample); }
\nNow, what do we do with that byte array? Well, as of Flash 10, our sound object, has a sample data event. When this event is called, when it needs new bytes, it'll call out to your custom sample data method.
\nWhat I do - and what you may choose to do, or not choose to do, is make my whole byte array first, and then read sequential amounts of data into the byte array each time it's called:
\nsoundBytes.position = 0; dynamicSound = new Sound(); dynamicSound.addEventListener(SampleDataEvent.SAMPLE_DATA, addSoundBytesToSound, false, 0, true); soundchannel = dynamicSound.play();
\nprivate function addSoundBytesToSound(event:SampleDataEvent):void { var bytes:ByteArray = new ByteArray(); soundBytes.readBytes(bytes, 0, Math.min(soundBytes.bytesAvailable, 8 * 8192)); event.data.writeBytes(bytes, 0, bytes.length); }
\nTo explain, about the 8 * 8192....
\n8192 is the maximum amount of samples you can use for each sample data event. However....each sample is a left and a right 4 byte float. So that's 8.....time 8192.
\nThere's tons of cool stuff you can do with this. If you don't believe me, look up Andre Michelle - he's THE MAN when it comes to this stuff.
\n", "date_published": "2009-07-15T00:00:00Z" } , { "id": "https://benfarrell.com/blog/2009-06-27-working-with-the-flash-sound-object-as-a-bytearray/", "url": "https://benfarrell.com/blog/2009-06-27-working-with-the-flash-sound-object-as-a-bytearray/", "title": "Working with the Flash Sound Object as a ByteArray", "content_html": "Last year I was experimenting quite a bit with Flash's computeSpectrum functionality. With computeSpectrum, I could take a snapshot of a playing sound and get volumes for various frequencies of the snapshot so I can visualize the sound.
\nNow, I wanted to take it a step further and visualize the sound before it's even played. Of course doing this with computeSpectrum is impossible because the sound has to be played first.
\nLuckily, in Flash Player 10, Adobe introduced sound.extract(); This can take the sound object and turn it into a byteArray. I came across an example by Thibault Imbert and of course the example worked very well to visualize a sound spectrum from a byte array - though I still wasn't sure what each byte represented in the Flash Sound object. And even though, the spectrum looked OK, a comment from Schell explained that this is a false waveform and to look at his blog post.
\nThe blog post provides some great working code - but there was quite a bit I didn't understand yet.
\nSo, I attacked the problem from another angle today. I thought - OK, I don't really understand the bytes that are coming OUT of the sound object, but maybe if I play with shoving bytes INTO a sound object to create dynamic sounds, I can get a better handle on tings.
\nThe real struggle came because I used Christopher Martin-Sperry's code to create a sound object from the raw byte array provided by an MP3 file. I used this before really looking into how all this stuff works. The code works fantastically - but put my brain on the wrong track.
\nI was on the wrong track because MP3's and Flash's sound Object work a wee bit differently from each other. In an MP3 file, there is a header to give some info about the file. Also, the MP3 file is broken into "frames". Each frame has a header to give information about the frame (bitrate and other things).
\nThis doesn't map so well to Flash's sound object. First of all, the bitrate of a Flash sound object looks like it doesn't vary. It seems to be 44.1 kilobytes per second all the time. And it's always in stereo. It looks like the source of the audio will always get upconverted or downconverted to this 44.1KBps stereo format.
\nNot only that, but there's no concept of frames. I thought I'd need to know how to read a frame header, or at least know how long each frame is to know what type of data I'm getting.
\nBut that doesn't appear to be the case. The byteArray extracted from a sound object appears to be an alternating left/right channel stream.
\nHow does the stream relate to time? I felt a little dumb when I figured this one out - because it's a little obvious. It generated a few tones before having this little gem of an epiphany.
\nWell, if the Flash sound object is 44.1KBps, that's 44,100 bytes per second. So when reading the stream, we read 44,100 bytes before a second is up. Right?
\nWell, not so right it would seem. You have to consider both channels. You'd read each channel 44,100 times to make up a second, making one second 88,200....something.
\nWhat is that something? Well my computer sciece teaching has failed me, but I can speculate. Adobe provides a morse code generation sample on their developer network.
\nIn it, there is a comment when they are writing the samples to the sound stream that they are writing 8192 samples to the stream. However, the length of the byte array they are actually writing is 8192 x 8, explaining that each sample is two 4-byte floating point numbers.
\nSo perhaps when reading floats with ByteArray.readFloat, you're actually reading both left and right floats from the same byte. If I'm right, then you're still using 44,100 bytes per second, but each byte represents two floating point numbers.
\nHowever, it works out, I've noticed that when creating six 1 second tones, by writing two floats at a time, 44,100 times, it clocks in at 6 seconds. So infer whatever you wish from that - I know I have.
\nMy next step is to figure out how to pull frequency data at a particular point in time.
\n", "date_published": "2009-06-27T00:00:00Z" } , { "id": "https://benfarrell.com/blog/2009-02-19-thewbcom/", "url": "https://benfarrell.com/blog/2009-02-19-thewbcom/", "title": "theWB.com", "content_html": "Just writing a short post to tout theWB.com which went live today after a makeover. My new team and I at Digitalsmiths produced the video players and the clip search tool. After just cancelling cable, I can't argue with even better video sites putting out great content. I'm always a fan of Buffy and Angel which they have plenty of. Everyone should also definitely check out the short WB shows "Joni and Susanna" and "Children's Hospital" starring Rob Cordry. Someday soon I'll definitely check out "Babylon 5", but I don't know if I have enough geek cred yet.
\n\n", "date_published": "2009-02-19T00:00:00Z" } , { "id": "https://benfarrell.com/blog/2009-02-12-music-inspired-gameplay/", "url": "https://benfarrell.com/blog/2009-02-12-music-inspired-gameplay/", "title": "Music Inspired Gameplay", "content_html": "This is the last part of my recent musings. Over the last two posts I've argued that digital storage has completely changed the way we listen to music, unfortunately in some ways for the worse. We're less inclined to replay an album in whole, and more inclined to put it all on shuffle. A friend of mine wondered how well Pink Floyd's Dark Side of the Moon would work today given our habits.
\nI wondered how we can increase replayability of music and get people to appreciate a song or collection of songs more and thought we can use games to achieve this. After all 30 year olds like myself grew up on video games with soundtracks that repeat every time you play a level. Why not use casual games to increase your band's audience - to get people to listen to your music over and over again and get the tune stuck in their heads?
\nOf course, its easy to slap a soundtrack on an online game - but how can we make the game and soundtrack work together for a great experience? How can we create musical escapism in games, just like MTV or musical theater? What is the gaming equivalent of breaking out into dance, or fast camera cuts timed to the rhythm?
\nAs you can tell, I've been thinking a lot about this lately, and I've recently read This is Your Brain on Music by Dr. Daniel J Levitin to better understand how our brains perceive music and how the various flavors of music appeal to us in different ways.
\nThe most obvious way to tie a game to music is tempo or rhythm. There are many rhythm based games on the market today - most noteably "Rock Band". Music has many more attributes, obviously, like the tone, contour, timbre, loudness, meter, theme, key, harmony and melody....not to mention that rhythm and tempo are two different things.
\nRelating any of these aspects to games can be challenging given where we are with mainstream realtime audio-processing technology. But even putting technology aside, how can we relate these to a piece of music in a systematic way?
\nLet's start with tempo. Tempo is the overall speed of the music. It's fairly constant for long periods of time - but can be an excellent baseline metric for establishing a musical pulse and to possibly tie it up with a visual pulse in your game. When people tap their toes, dance, or bop their heads - its generally to this tempo, or the beat of the song. So tying visual elements to this beat is probably the most effective tool I can think of.
\nAnd this is good, because a beat can be measured in pop music by picking the range of frequencies that the bass drum is on and listening to the volume of this frequency over time. So tying visual elements to a beat is a very real tool that can be done automatically in music game creation.
\nTempo only goes so far though. It's a very predictable musical trait. Levitin argues that our musical tastes can somewhat be centered around complexity and predictability. Children, for example, enjoy very simple and predictable music. As we get older, and the more we listen to music, the more boring this simple and predictable music becomes. So we listen to more complex music - but constantly strive for the right amount of predictability and the right amount of complexity. As we listen to more and the level of complexity of our favorite music will probably go up to. This is one of the reasons that the music of other cultures can be distasteful, or classical music, or jazz can be distasteful for people with pop only listening habits. The more unfamiliar we are with music, the less predictable it is. And if its not predictable at all - as another culture's music can be to us - then it can really be unpleasant.
\nSo, going beyond the beat to capture other nuances of the musical piece should be very important in manipulating visuals or gameplay elements. ONLY syncing a predictable visual beat with a complex soundtrack would be such a shame because you lose all that which makes the music meaningful - and you can lose the connection between the two. Alternately - having lots of movement that has nothing to do with the flow of the piece but loosely tie to frequency at a particular time, can create a similar disconnect because you're picking up unpredictable parts of the piece to tie them together visually with the music.
\nSo what types of musical nuances can we pick up and use?
\nRhythm is another form of timing like tempo but it's how notes are grouped together into phrases. Guitar Hero and Rock Band use rhythm effectively. I believe Harmonix actually does this by transcribing the notes themselves for each instrument, and grouping them together to have you play out a phrase at a time. They're not picking out random notes from the guitarists score and having you play them - no, they're taking the most meaningful notes that make sense to put together the phrase with the limited amount of notes you can play.
\nTone and frequency for example are aspects that may be quite hard to integrate effectively. Considering an overall musical piece, many instruments are playing at different pitches and frequencies all at once. The bass guitar has a very low frequency, while a flute can have a very high frequency - even though both can be playing the same note or different notes. Taken out of context, different frequencies/tones in music don't make us tap our toes.
\nPitch, harmony, contour, key, and melody - taken together can be an entirely seperate but equal way to tie visuals to your music. Unfortunately - it can be very difficult (if not impossible) with mainstream game technologies like Flash to take this into account in an automatic way. I hope I'm wrong - and I hope somebody PROVES I'm wrong and lets me in on the secret, but consider this....
\nUsing Fast Fourier Transform methods (FFT), I can grab the volume of any frequency at any time. Can this provide me with what note is playing? Well, maybe....but only if a single instrument is playing. Unfortunately many instruments are playing at many different frequencies, and many can bleed into (if not use) another's frequency. I know some smart computer scientists have developed pitch detection - but it hasn't made it's way into any code libraries I know of yet (though I should look beyond Flash to see what I can find).
\nAlso, on a technical level, using MIDI files, if the music is transcribed correctly (or if it originated on a sequencer or a computer in the first place), we would have access to all the notes seperated out into the different instrumental tracks.
\nSo what if all this technical stuff presented no barrier? How could we use it?
\nOne of the most interesting parts Dr. Levitins books for me was Appendix B: Chords and Harmony. Music, if you look at it in a very dull light, is all about patterns and manipulating people's ears by breaking in and out of the predictiability of those patterns.
\nTalking about pop music, there is a verse, a chorus, and a bridge. A verse is normally a melody and tempo played over a few times. At the end of the verse, we get into a similar thing with a chorus, which is generally shorter. Going from chorus to verse breaks one pattern, but picks up another musical pattern which uses slightly different chords with a slightly different tempo. So predictability is broken, but not very much.
\nAnother way patterns are established is with long standing musical tradition. The blues is somewhat defined by going from I Major to IV Major to I Major and then IV or V major, and then back to I Major - this happens to also be the basis for rock music as well. Now, this is another established pattern - and when we break from this pattern, we're introducing unpredictability.
\nThe most intersting pattern Levitin mentions is when a chord is either resolved or unresolved. In Western music, our ears have been trained to consider a tritone or an augmented 4th, the most unresolved interval we could possibly hear. In fact, as Levitin recounts, this interval was banned in the Catholic church and named "Diabolus in musica" citing this interval as the work of the devil. On the other hand, a simple major chord is considered resolved. So when if we play a chord containing a tritone, we expect, and almost demand that it be resolved by something like a major chord.
\nWhy do I bring these up? Well, this type of musical behavior can start to paint a more visual or motion based picture. When music breaks expectations either by changing tempo or with different chords, or even moving the notes up or down an octave - likewise our gameplay elements should match this level of broken predictability.
\nLikewise, when a chord is left unresolved - so too should our visuals on screen. You know something is going to happen, but you don't know what.
\nTo spell it out more clearly, we can establish a visual and animated baseline in our games by listening to the beat of the music, and timing elements to this. Assuming a pop song, our baseline is a song verse.
\nDeviating in minor ways while the verse is played, maybe be changing the tempo, going up an octave, or different instrumentation, will produce a minor change in the visuals or animations.
\nGoing from verse to chorus however is a less subtle change, and often implies changing the tempo, chords, or otherwise quite drastically. In this, the gameplay needs to change in the same fashion.
\nHitting a chord that needs to be resolve creates musical suspense - and so too should it create visual or gameplay suspense. Something on screen about to fall, or teetering from side to side.
\nGoing back to technical implementation, however, assuming we had access to the entire score of notes in a musical piece, can we feel out these suspensful moments or feel out when a song is predictable and when it becomes unpredictable?
\nI believe the answer is yes, but it would take quite a bit of work to look for augmented fourths, or other unresolved intervals, or to run pattern matching algorithms on our melodies.
\nA more pragmatic approach might run a FFT analysis to find beats, and possiby even do some light pattern matching to seek out changes in tempo. Meanwhile, do some manual markup of subtle changes in verse, and then manually mark where verse goes to chorus, chorus to verse, and where the bridge is. We can also manually mark where we're creating suspense, and then where we resolve that suspense.
\nIn this fashion, we could create a sort of music markup language to map a timeline of all these events and use this to create a musical gameplay experience.
\nI guess my work is cut out for me!
\n", "date_published": "2009-02-12T00:00:00Z" } , { "id": "https://benfarrell.com/blog/2009-02-01-weekend-project-bye-bye-cable-tv/", "url": "https://benfarrell.com/blog/2009-02-01-weekend-project-bye-bye-cable-tv/", "title": "Weekend Project - Bye Bye Cable TV", "content_html": "Last Thursday my Hauppauge Dual Hybrid HDTV Tuner card came for my computer. Until then I'd been using another dual tuner card to record and watch cable TV through Windows Media Center. It was cool and all, but in that time we've gotten Netflix, and found out about Hulu.com, and I got a job working at bringing video and video search to the web from major studios and content producters. So there's a ton of digital media out there on the web (stuff you might actually want to watch.....stuff that you might normall have to have cable TV for).
\nNetflix On Demand is damn cool - I was watching HBO series and new movies through a nice Windows Media Center interface called MyNetflix.
\nNot only that - but the digital content from online looked a lot better than my cable signal did (I still have a normal non-HDTV by the way). I think that they compress the content digitally before it comes through the cable wires - and then of course Windows Media Center compresses it again in a different way....and thats no good when you compress the same digital video twice under two different compression standards.
\nSo - thus begins my experiment: Bringing HDTV to my computer even if my TV doesn't really support it.
\nSaturday morning, I installed the card. I was sorta hoping that it would just magically pick up HDTV from the air without plugging in any antennas or anything. Well that was a big fat no.
\nThen I went to Best Buy - I got an multidirectional amplified indoor antenna. Basically, a big flat sqaure, probably with some wires inside, and you plug it into the wall. So - I hooked that up....in my finished basement...where we watch TV. OK, a big fat no.
\n"This is discouraging" I thought - but thought I'd keep trying, since I was in the basement after all. I grabbed a roll of coax wire and ran the indoor antenna outside to my raised porch. I tried again. This time it was more promising. I scanned through the channels, it said there were a few digital channels available - so I started flipping through.
\nIt looked good but not great. It was a little snowy, some bad pictures, definitely not looking like what I expected out of HD. Oh but what's this.........there's a WRAL-DT? Umm....hey a whole bunch of DT channels in the low 1000's. So apparently, I wasn't looking at HD, Windows Media Center just threw in the Analog channels into my guide for good measure at the beginning of my guide. OK, lets flip to the HD stuff.......daaaaaaaaaamn. It looked awesome. Very crisp and clean on my computer monitor, I hadn't even hooked it back into my TV yet.
\nSo, I set out to see what channels I could get.......
\nABC, check....NBC....check, CW....check, PBS....check, FOX....check, CBS.......where's CBS?
\nI went to my local CBS affiliates website, and found that I'm about 15 miles outside of their range using an indoor antenna.
\nSunday morning, I set out to the hardware store to see what else I could buy in antennas. I sorta wanted an outdoor one, but the thought of me going up on the roof with my non-existent ladder and installing something - somewhere, somehow on my house seemed hazardous to my health.
\nI got to Lowes, saw they had rabbit ears and outdoor antennas. Thats it. But I picked up a 4 foot long outdoor antenna box and looked at the packaging - "Can be used in your attic". Hey! Awesome, I hadn't thought of that! I can put this huge freaking thing in my attic and run a wire down 2 stories to my basement....somehow.
\nSo I bought it at $60 - and a 100 foot roll of coax cabling.
\nI got it home, brought it into my attic, started unfurling it - and I won't lie - its very unweildly. Picture me, in an attic with no floor, standing on beams trying to shift around a 5 foot wide, 4 foot long antenna. I dropped it in place eventually - didn't bother bracketing it to anything, and connected the coax cable, and searched for a way to get it out of the attic.
\nAnd look....a vent, with a screen over it thats slightly unhinged at the bottom. Perfect place to drop cable out. So the cable was unreeled, and unreeled, and unreeled. I brought it in around the window entrance for the cats into the basement and over to the TV.
\nAt long last, I have all the channels I need! CBS included, so now it's just a matter of returning the indoor antenna to Best Buy and setting things back up on Windows Media Center. Media Center tells me I have 21 channels - most don't come in, but as I look up the call letters on the internet, I find that those are pretty far away like in Wilmington, Winston Salem, Greensboro, etc.
\nI settle on 8 channels - WUNC (PBS), WRAL (CBS), WTVD (ABC), WNCN (NBC), WRDC (CW), WUVC (Univision), and WRAZ (Fox).
\nNow the tricky part....for the short time I'm actually paying for cable still, I want to make use of it, and Windows Media Center (as far as I know) won't let you have both over the air and cable channels together set up at the same time. So, I'm installing Beyond TV as I write this (a different DVR software solution), and I plan to use that to record cable, and Media Center to record over the air. But that's just till we cancel cable. I mean - I'll record Battlestar as long as I can until I have to go to SciFi.com (or someone elses house) to watch it.
\nBut at last count - the Daily Show and the Colbert Report are free at Comedy Central.com. Monk and Psych are free at Hulu.com, my wife watches Ghost Hunters and both of us Battlestar which is free at Scifi.com. I just can't think of any other cable shows we watch really. And if miss something, we'll wait and get it on Netflix. And hopefully I'll get an invite for Boxee TV on Windows (please please please) since I filled out the request form on their website last week.
\nSo now, I'm just waiting for Beyond TV to install, and then go hit a superbowl party even though I hate football.
\n", "date_published": "2009-02-01T00:00:00Z" } , { "id": "https://benfarrell.com/blog/2009-01-26-consumption-of-music-on-demand-part-2/", "url": "https://benfarrell.com/blog/2009-01-26-consumption-of-music-on-demand-part-2/", "title": "Consumption of Music on Demand Part 2", "content_html": "In my part one of this post, I took a look at how digital video and audio recorders and portable devices have changed the way I consume and perceive media. What I'm most interested in is music - and I discussed that because of the ability to hold all of my music library in one place and the ability to get music from many sources free on demand, I've started listening to music on a very superficial level, and don't give albums concurrent repeated listens.
\nIf I have this problem - others probably do to, and may even be perfectly happy to be oblivious to this.
\nThis is also a problem for musicians. If people give music only superficial listens, and are just as happy to move onto the next song by a different artist, why would a musician have fans that bother coming to shows, buy t-shirts, or buy follow-up works by that musician? Isn't it easier just to turn on an internet stream and listen on shuffle than to seek out and buy an album or see who's playing in your town on Friday night? It's as though being a fan is becoming a lot less fanatical.
\nMy concern over this is that music will be written to appreciate on a superficial level - will only make you nod your head to a beat. Of course it's already been happening in pop music for years. Musicians will only get their "radio friendly" hit song played wide-spread. But many times, this radio-friendly song will be the gateway drug that leads you into the album, and get you to repeatedly listen to all songs by the artist. Other songs on the album could provide deeper enjoyment.
\nWhether this problem is new and exasperated by our digital devices, or an old problem but just becoming apparent to me as I change my listening habits, I have to wonder what we can do to change these habits and produce ways to make people want to listen over and over again and come to a deeper appreciation for the music and the artist. Listening to music over and over again can even make things more memorable. How many times have listening to a song for the first time in years bring you back to old times when you were listening to that song constantly. It can be very nostalgic.
\nI've been thinking a lot lately about music and video games (as you can infer from my blog posts). I think that games like Dance Dance Revolution, Guitar Hero, Rock Band, etc do go quite a way in invoking repeated listening habits in music. Pressing buttons in time to a beat, however, is just one direction to take this - I would call this a music creation game (even though you aren't actually creating the music, there's the illusion that you are). I'm quite interested in exploring a game where music is creating the game play environment that you're in.
\nAnother way to put this is I'd like to explore what games can be to music as what music videos are to music. The best music videos, in my mind, produce another world where people break into singing and dance. A mailbox will spring to life on the street and dance with Bjork. Christopher Walken will dance his way down an escalator and fly through the air in grand choreographed moves. Weezer will give a show to screaming members of the Happy Days cast. You get my escapist point (and my Spike Jonz fixation). I'm not a musical theater fan, though Gilbert and Sullivan have a lot of experience with this too, and they go a little farther back than MTV. From what I understand, even the ancient Greeks had musical theater.
\nBringing it back to games - a lot of great work has been put into cinematic gaming soundtracks, but to my knowledge, they are just soundtracks and don't really change what happens in the game. A change in tempo doesn't make more bad guys come out to hurt you, a minor chord struck doesn't usually signify coming doom. I shouldn't say never. I'm not an avid gamer - I just hadn't ever heard of this attention to music detail. In fact - it's quite the opposite - more bad guys coming out will change the tempo of the music - and coming doom, will change the music to a minor key. Game composers will typically write tiny segments that can be switched depending on what happens in the gameplay.
\nSo, how can we produce a truly musical game? One that gives a level of escapism straight from Broadway and MTV, gives music a forum to be listened to again and again, and is fun? One where the music isn't changed for it's gameplay, but where gameplay revolves around the music?
\n(Yah, I'll answer that question next, or at least try to)
\n", "date_published": "2009-01-26T00:00:00Z" } , { "id": "https://benfarrell.com/blog/2009-01-12-consumption-of-music-on-demand/", "url": "https://benfarrell.com/blog/2009-01-12-consumption-of-music-on-demand/", "title": "Consumption of Music on Demand", "content_html": "Tivo and iPods (or more generally a digital audio players and a digital video recorders) have changed the way we consume media. It's a pretty obvious statement I know - but hear me out. I want to talk about music, but let me first say something about tv/film.
\nI was a late comer to both Battlestar Galactica and 24. I didn't start watching either one until a couple seasons in. I recorded a bunch of 24's with my DVR, and bought a few seasons of Battlestar Galactica on DVD. Either way, I wound up watching entire seasons of a show in a week or two in contrast to the six or nine months it usually takes to watch a season of a show.
\nWatching habits were changed from when the network demanded it to whenever I wanted to see it. In this, you could almost argue that the story itself behind the show has changed. There's no more downtime between episodes to think and ponder, and be on the edge of your seat about what happens next.
\nIn fact, my DVR, Netflix, and DVD collection have changed my attention span on shows when I do have to wait to access them. When I was watching my 24 marathon, I was really into it. Then, when I had to wait a week to see what happened, I was less into it. And now, I'm not even sure how much I care about 24 now that its been a whole year.
\nThis is stark contrast to how we used to watch TV. Either we were chained to our sets when our favorite shows were on, or our favorite shows just happened to be those that we had the specific nights free for.
\nThe point is that having constant, on demand access to TV and film can ultimately change the story as we percieve it. Watching TV can become more like reading a book. You can rewatch/reread, read/watch when you want, and you don't have to read/watch the whole chapter/episode in a sitting, or you can finish the entire thing in one sitting. This is especially true as we can watch video on our mobile devices now.
\nMusic can be a little more complex. In addition to being short form, where consumption of a song can take only 3 or 5 minutes, there's not usually a cliff-hanger or a compelling reason to listen to the next song on the same album for most people.
\nThis became apparent when your single disc CD player became a 5 disc CD changer back in the 90's. I found it was easier to put my 5 discs on shuffle to get a little variety in each music listening session.
\nMy car was a major place that I'd listen to music, and it was especially great to have my music on long rides. My car, however, had just a single disc CD player. That means when I put a CD in, it would stay in. And because I was driving it, was a bit complicated to remove and change it unless there was a break in traffic. I also only had a limited selection to choose from, as I'd usually only take the last 5 or 6 CD's I bought with me on a normal car trip.
\nThis means when I bought new music, I'd get very familiar with it. I'd listen to it over and over again.
\nI suspect that most people's music habits had the same limiting factors. Either you listened to what the technology at the time allowed you to hear, or what the radio allowed you to hear - and each of these things led to the same thing, you'd listen to a song over and over again.
\nOf course with digital audio players with lots of storage this has all changed. I have a Microsoft Zune with 120 Gigabytes of storage capacity. This means that I can put my wife's music collection and my music collection (probably 500-600 CD's) on my Zune in its entirety 3 or 4 times over. Many music players have at least 1 Gigabyte of memory allowing 10-15 albums at a time.
\nSuddenly, it becomes more convenient just to put every song you own on shuffle than to find the last albums you bought to listen over and over again - and given people's tendency towards variety, a big shuffled list sounds more appealing when it comes time to listen to music.
\nSo, listening habits have changed - and so has how we percieve music as a result.
\nHave you ever listened to an album or song, and didn't really like it at first, but maybe after listening to it for the 5th time or the 10th time it's one of your favorites? It used to happen to me all the time. But it really doesn't happen to me much anymore. Music listening has become sort of superficial to me. I'd buy some new music, listen to it once, and then it appears on shuffle sometimes as I'm buying newer music.
\nIt's somewhat of a personal goal to give my new music a better listen from now one, despite the fact that my entire collection fits in the palm of my hand.
\nSo as things are getting long, I'm going to continue things in a second post where I think about how people can listen to music on a less superficial level despite the technology we have.
\n", "date_published": "2009-01-12T00:00:00Z" } , { "id": "https://benfarrell.com/blog/2008-11-28-popfly-reactive-music-game-prototype/", "url": "https://benfarrell.com/blog/2008-11-28-popfly-reactive-music-game-prototype/", "title": ""PopFly" Reactive Music Game Prototype", "content_html": "It's been a busy past month just getting a new job and a new car - I felt like I didn't have time for anything. Well, it just the Friday after Thanksgiving today, and I feel like I have all the time in the world this coming weekend, so I'm starting to get excited about my latest pet project.
\nI've been busy early this fall with writing "reactive music games". If you read my blog, which you probably don't, you'd know that I couldn't shut up about music games all summer. I even did a little research into the different types. The reactive game, is where music affects the things that happens in your game environment - but you don't have to play along to (like Guitar Hero or Rockband).
\nThere's actually no popular music games I can think of that ARE reactive. But I like to think of reactive games really.... like an interactive music video. I even watched a bunch of Spike Jonz videos before I started sketching. I have 10-15 games to do in the short term, and most are designed to be pretty easy to develop and generic - ESPECIALLY since this summer I've been working on min-framework in Flash AS3 that listens to beats in music.
\nThe first game, I've done is basically an alpha, and if the music doesn't load, just hit refresh (sometimes my security sandbox seems to go a little wacky). I've codenamed it "Popfly" in my sketches, cause the balls kinda pop and fly out with the beats.
\nI've tried to pay special attention to things that I've noticed I didn't like in other games. Specifically, trying to tie visual events to music better. Balls pop out, and when they reach their peak, its not very tied to music because it's a split second after the beat. This is why I put a subtle glow burst at the bottom when the balls do pop out. I also gave the side gutters some ambient glow - just to have something additional on-screen that's tied to the music in the game. I also made some bigger balls pop up when the volume gets high - and it nicely adds a little crazy mosh effect when the music goes crazy.
\nI can't wait to get through some more games, and see where this takes me, and hopefully way down the line, do a nice in-depth game with some characters.
\nBut for now - here's the PopFly game
\n", "date_published": "2008-11-28T00:00:00Z" } , { "id": "https://benfarrell.com/blog/2008-11-28-bizkidz/", "url": "https://benfarrell.com/blog/2008-11-28-bizkidz/", "title": "Bizkidz", "content_html": "Or should I say....BIZKIDZ.COM! The last site I worked on at 360KID finally went live (and out of beta even)! Its been a long time coming, I think we started a year and a couple months ago? I've since moved on and taken a new position at a video search company called Digitalsmiths here in RTP. The reasons are numerous, though 360KID is still an awesome company, but I just had to move on to new challenges and other people to learn from.
\nAnyway, where was I? So BIZKIDZ.COM is a site for kids to go and set up a storefront, and sell actual product from real online retailers. Lots of retailers in fact, at least like 400 of em last I checked. The idea was to create a virtual lemonade stand of sorts. As a child with an account, you'd populate your store with products and decorations, and send links to family and friends. If they buy anything through the store, the account holder gets commission!
\nIt's pretty cool - I ended up doing most of the front end. We got some help from a great contractor - Clint Little, and J2 Interactive did our server side architecture. What was great was that I got to touch every part of the project - I did 99% of the Flex (sorry Clint, but those initial designs we had you work on, I had to do them over when the client changed their mind, oh well....great job though!), and maybe 5% of the server side stuff, and most of the HTML. And of course our talented art team at 360KID did some wonderful designs and illustrations.
\nI got a lot of Flex training on this, how to make a Flex application NOT look like a Flex application - all in all I think it was a job well done - and if I know BIZKIDZ, they'll be continuing developing their site with 360KID to make it even better than we imagined at the beginning.
\n![\"bizkidz1.jpg\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2008/11/bizkidz1.jpg\")
Like many of these heavily animated and interactive games, Magic Numbers was developed while working for 360KID.\nIf I did a better job of keeping track of my work, I could say more about my role in developing this, but for now, I'll\ncopy the description from the 360KID portfolio website:
\n**When conceiving this math and number sense game, Sesame Workshop decided to bring back an old favorite character from yesteryear who hadn't been visited for a while: Mumford the Magician.
\nWhile Mumford was a perfect character to help host a game about counting and simple math, it did present a special challenge: no official digital line art version of Mumford had ever been created. Using style guide assets for other Sesame Street characters as reference for lines and shapes, the 360KID team was able to create a digital version of Mumford that sits easily beside characters whose 2D animated looks were previously well defined.
\nOn top of that, the game itself is a huge hit.**
\n![\"magic](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/portfolio/sesame_magicnumbersa.jpg\")
![\"magic](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/portfolio/sesame_magicnumbersb.jpg\")
![\"magic](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/portfolio/sesame_magicnumbersc.jpg\")
I've been playing a bunch with physics engines for the last few weeks. Physics engines basically serve to give your Flash Sprites some mass and body. Your physics world can update your graphics so they can bounce off each other, bounce off walls, fall to the forces of gravity, and succumb to other forces all in a semi-realistic manner.
\nMy exploration first took me from Papervision3D to WOWEngine. Papervision3D is, of course, the popular realtime 3D engine for Flash. It doesn't have anything to do with phyics - but WOWEngine seems to be the hot dicussion for bringing physics into your 3D world.
\nWOW works like this....
\nImagine a 3D world that you never see. Yep, thats it. That's what the WOW does. It's the 3D world that never gets visualized in any way shape or form. Of course, the 3D world in question has complete information about all objects in the world and how they interact with their surroundings and forces surrounding them.
\nIt's actually a nice little setup - you get to keep your Papervision3D or regular 2D world separate from your physics world. So you have 2 different worlds - each operating independently of each other. The way you tie them together is with a single handler ENTER_FRAME handler where you cycle through all the physics bodies and update your display with the proper x, y, z, or rotational properties. The 2 worlds make it easy because you could use any 3D engine, 2D engine, or whatever engine you want and glue them together - there's no dependencies, no skinning, no nothing - so it's pretty handy, and a quick concept to grasp. 2 worlds - gotcha!
\nThe problem is that the WOWEngine isn't that great of a physics engine (yet!). Another unfortunate problem is that its the only Actionscript Physics engine that handles 3D. So if you use 3D, you're kind of stuck using this or writing your own.
\nWOWEngine is based off of another engine called APE (Actionscript Physics Engine). APE is a 2D physics engine that seems to actually be pretty good. The problem with WOW, is that it has such potential, and works pretty well, but hasn't implemented the richer features of APE yet. The author definitely plans to, it's on the roadmap, but I'm impatient need it now!
\nThe killer feature that makes a great physics engine in my opinion is Rigid Body Dynamics - which APE has and WOW lacks. To explain rigid body dynamics a little bit, let me tell you a little bit about my initial experimentation.
\nMy big "hello world" application was basically making a few balls drop from the sky and hit the ground. I initially tested in 2D, and just ignored the Z axis. So we're talking normal Flash Sprites here. So the balls hit the ground, bounced off of each other, all pretty cool - and what you'd expect from a physics engine.
\nNext I tried the same thing with squares. Still cool, but not quite right. That's because I still had to treat each square as a generic ball particle. With no Rigid Body Dynamics - I couldn't build a little fortress with my cubes! That's because without being able to define a polygon or even a cube, whatever graphics you have.....just sort of act like balls and slide off each other.
\nSo that's the my big problem. Someday WOW will be awesome, but not yet for my purposes.
\nNext I checked out Foam. Foam is a 2D Physics engine for Actionscript 3. Foam is actually pretty sweet! I didn't spend long on it, though. The reason is that after I experimented with WOW, I didn't really like how Foam worked. It seemed to handle Physics well - but it seemed to integrate the graphics and physics world too much for me. I think I preferred them seperate - especially in a situation where maybe I want to use a different graphics engine. I didn't get far into Foam, but it seemed more like that Foam would take over your stage and you'd skin the physical bodies. Before I went into this physics thing, it's how I expected things to work - but I just couldn't dig in after playing with WOW.
\nSo again I moved on...
\nFinally I hit Box2DAS3. This is the one I settled on - it has it's problems, but if you can overcome it, then I do think it's the best. Box2DAS3 is a Actionscript 3 port of Box 2D. Box 2D is an opensource physics package for C++. In fact, Box2D has evolved to become Bullet - a 3D version of the physics engine. Hopefully we'll see Bullet 3D someday soon.
\nBox2DAS3 supports all the great features a good physics engine should....well OK OK, I'm not an expert - it supports all the great features I *think* a good physics engine should. It also follows my beloved 2 worlds philosophy that I liked from WOW/APE. For me it's perfect....
\n....well almost perfect. I think it has one flaw. The flaw is that it's a C++ port, and not really specific to Flash. If I went back in time and warned myself of this, I would've been pretty dismissive and said "So what? Code is code!".
\nWell first of all it seems like classes have the bare minimum of comments - so you just kind of get the picture what everything does. And it's really not clear what each class or package does - unless you know....um...how to write a physics engine. So it's a little bit of a catch-22. Like a simple body is name b2Body. To create a body you need to create a shape definition (b2ShapeDef), convert that to a shape, and then convert the shape to a body, and then add the body to the world. And it seems there's only one way to add a body to a world, and that's through a reference to the world, even though when you create a body from scratch, there's a world parameter - but it doesn't seem to work. Well, I'm rambling. Needless to say it's confusing for a newbie - and on top of that there are things like manifolds and AABB - all things I don't understand but there to get in the way and hamper learning.
\nBut, if you study the examples they give you - you start to get a feel for how things work. And once I did that - I stumbled a little, especially on design patterns - but got things working pretty well. My design pattern quandry was that the framework was very incompatible with an inheritance based design - it was really going against the grain when I tried to use inheritance in my design. What you really need to realize, is that a Factory type of design pattern works much better.
\nI ended up creating a base class like you would in Papervision3D. The base class extends the stage and creates a common physics world setup as part of that stage. Your main class would inherit this "physical stage" and contain the main stuff you want to program (hopefully keeping references to b2bodies and b2worlds to minumum, just cause it's hard to understand!).
\nOn the side - I have a physical object factory static class. Here's I'd call createPhysicalBall or something like that, and poof, it'd create me a ball....passing back seperate references for the graphics and the phyiscal body.
\nSo that's how I roll with the physics. It adds some great realism to the boring old games.
\n", "date_published": "2008-09-19T00:00:00Z" } , { "id": "https://benfarrell.com/blog/2008-09-14-visual-ambiance-in-music-games/", "url": "https://benfarrell.com/blog/2008-09-14-visual-ambiance-in-music-games/", "title": "Visual Ambiance in Music Games", "content_html": "So it would appear that visual ambiance in music games can be pretty important. Take most rhythm games...
\nBackground colors change, flashes appear, etc, all in time to the music in games like Guitar Hero and Rockband. In PaRappa, characters dance and jerk their bodies in time. What you end up getting is more music video like.
\nRhythm games, those games where you have to tap your fingers, feet, or guitar to the beat as part of gameplay can feel more music video like because you end up "playing" the music to some effect.
\nReactive games on the other hand, may have to rely on visual ambiance more. Reactive games, are more often than not, played like any normal game. The environment in the game may react to music, but you don't. So if you're not required to react to the music, the music can start to take a back seat to the game play, and it starts feeling less choreographed.
\nThis is why visual ambiance can become more important, and probably should be in your face more in a reactive game. It would seem that when you don't control the choreography of the game, other musical visuals need to step up to help out and make the game FEEL musical.
\nI used the example before of AudioSurf (published by Valve). I crticized it for not tying the visual to the music as well as it could have. I can't criticize it for attempting ambiance to tie the music into the game better, because it does an adequate job of this with colors and background visuals. However, the environmental reactions to the beats still seem too out of touch with the music, and the ambiance is all it has going for it to make it feel choreographed.
\nSo, the closer you can tie in your reactive environmentals the better. But like I said, even that might not be enough, as your game starts focusing on the goals of the game - which is why visual ambiance becomes a VERY important silent partner.
\n", "date_published": "2008-09-14T00:00:00Z" } , { "id": "https://benfarrell.com/blog/2008-09-01-scrapbookcreator/", "url": "https://benfarrell.com/blog/2008-09-01-scrapbookcreator/", "title": "", "content_html": "title: "Scrapbook Creator"\ndate: "2008-09-01"\ncategories:
\nThis was another project that I think was a proof of concept and not for any particular\nclient. Kids could create a flash/web based scrapbook by dragging and dropping images, stickers, and text onto\npages of their book.
\n![\"scrapbook](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/portfolio/ScrapbookCreator.jpg\")
This was a quick little freelance Flash project. Basically, there was software (maybe it was Zinc)\nthat could convert Flash apps into screensavers. In this one, we basically just had shoes fall from the\nsky and land on the ground.
\n![\"DSW](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/portfolio/DSWShoes.jpg\")
\nDSW Shoes Screensaver
OK, I will be very interested to see how this plays out. The site is Music Video Games. I'm so curious in fact, that I'm analyzing and critiquing the SMALL amount of website they have up.
\nSo, what I gather from press releases, is that Music Video Games will be a portal to deliver casual games with beat matching to popular songs (record label Rhino is getting involved).
\nKind of cool, not quite the angle I'm thinking I'd like to go, but pretty dern close. Like I said, it's one to watch once it gets going, but right now I don't have much to go on other than....
\n1. Its lame logo... whats up with that? If that's the extent of the design skills, I'm not sure if it's gonna be anything special
\n2. It's name - OK the spin with MvG, and comparing it to MTV, not bad - but why not just call your company MvG, the logo designs itself practically!
\n3. Patents - I don't know what kind of "patented beat matching" technology they came up with, but I've put a lot of thought into finding beats, and if it's anything beyond the obvious methods I'm employing in my experimentation, more power too them, but if not, I'm going to be very depressed for the world if they land a patent.
\n4. The plan - A casual gaming portal? I'm not sure if they'll land a wide audience if they take some games and add music without a bigger hook. I mean, its pretty cool and all, but once launched and the novelty wears thin, I wonder if they'll really have much to differentiate themselves from a normal site. Yah, each game will have lots of replay value due to songs they could mix and match if they're smart - so we'll see. This is not really a criticism but curiosity I guess.
\nSo, will this be successful? Will we see innovative games? Or will we see a bunch of rhythm games or guitar hero clones with an impressive music library on one site.
\nOnly time will tell, I wish I wasn't teased with the crappy splash page. Oh well.
\n", "date_published": "2008-08-22T00:00:00Z" } , { "id": "https://benfarrell.com/blog/2008-08-08-reactive-music-games/", "url": "https://benfarrell.com/blog/2008-08-08-reactive-music-games/", "title": "Reactive Music Games", "content_html": "Well, I've been doing some research lately. Back in January I wrote up a a blog post titled "It's been 10 years since PaRappa - now what?". I was exploring the mainstream music games. Little did I know that these games are all part of the group known as "Rhythm" games. Of course, many folks know what a rhythm game is, and I do too.
\nWhat I DIDN'T know is that Rhythm games are just one of a handful of different types of music games. Rhythm games can include any games that play a soundtrack and must press buttons in time to the game. Examples include "Dance Dance Revolution", "Guitar Hero", "Rock Band", and "PaRappa the Rapper".
\n"Pitch" games recognize the pitch of your voice from a microphone, and like Rhythm games, you must keep time with the music, but instead using the pitch of your voice to sing along. "Karaoke Revolution"is one note-able example. "Volume" games include games like "Mad Maestro" and the upcoming "Wii Music". In these games you can control the volume of the music in the game through various means, thereby influencing the game play.
\nNext are "Eidetic" music games. These games will play a musical sequence, and it's up to the player to repeat the sequence back. In this regard, "Simon" is the oldest music game around.
\nTo be short and sweet, I'll lump "Free Form" and "Generative" music games together. In both of these game types, the player and the elements of the game come together to create a unique sound track. It seems that creativity is more likely in the "Free Form" music game.
\nThe last genre, and the one I'm most interested in, is the "Reactive" genre. Here, a pre-composed soundtrack plays, and in game elements can react to the volume, frequency patterns, and beats in the music. All of these genres, with the exception of the Rhythm game, are hard to come up with examples for.
\n![\"AudioSurf\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2008/08/audiosurf.jpg\")
That said, there are a handful of Reactive music games that I'm aware of. The first was published in February 2008 and created by an outfit called "Invisible Handlebar" and available from Valve software's Steam game portal. In this game, called AudioSurf, you move a race car left and right over a racetrack. The goal is to collect blocks that appear on the road, and stack them in groups of 3 to get points. You load in your own MP3, and the background environment animates in time with the music. The track itself dips and rises in time with the music as well. Supposedly, the blocks you collect also appear in time with the music as well - however they appear on the road far in advance and far in the distance of when you collect them. As a result the gameplay doesn't really feel in synch with the soundtrack. It's only if you really pay attention to your surroundings, and not to the game play to really get that connection. For example, a hip-hop track by "Jurassic 5" is a little more bouncy than a rock track by "Living Colour", but it can get hard to tell in gameplay the difference between the songs.
\nThere are a few Adobe Flash examples of this like "Rhythm Night", but the game play isn't all that great. Most of the Flash examples to date are either Rhythm games, Eidetic, or just general music lesson games.
\n![\"Pteranodon\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2008/08/pteranodon.jpg\")
Fortunately there is an excellent example to draw from - and that's "Pteranodon" by an independent game developer from Sweden known as Nifflas. His name is Nicklas Nygren, and "Pteranodon" is a classic side-scrolling space shoot-em up game. The twist of course, is that the enemies appear on screen and fire at your in time with the music. Its a pretty simple game by today's standards - though the connection between what's on screen and the timing of the music makes me giddy. Apparently he composes his own music, so it's no surprise that "Pteranodon" just features one track, but one can just imagine where you can go from there.
\n", "date_published": "2008-08-08T00:00:00Z" } , { "id": "https://benfarrell.com/blog/2008-05-27-a-codie-award-to-add-to-my-resume/", "url": "https://benfarrell.com/blog/2008-05-27-a-codie-award-to-add-to-my-resume/", "title": "A Codie Award to add to my resume!", "content_html": "The 2008 Codie Awards have been announced. Little did I know that a project I played a major part in was at least nominated for a couple Codies, we actually WON the award for "Best Instructional Solution for Students at Home". Pretty nifty, and its just the latest in a series of awards for the Pokemon Learning League.
\nSo far the awards have been the Codie, a 2007 Outstanding Products Award from iParenting, a 2007 Best Educational Software Award, and we were a finalist for Distinguished Achievement from the Association of Educational Publishers.
\nI won't go into the "making of" in the least, cause you know, it's safe to assume that most clients don't appreciate any ups and downs of working with them made available in a public forum. But.....pretty nifty anyway right?
\nThe project is: http://www.pokemonlearningleague.com/
\nAnd of course, yes I played a good part in it, but there were plenty more people involved at the company I work for 360KID.
\n", "date_published": "2008-05-27T00:00:00Z" } , { "id": "https://benfarrell.com/blog/2008-05-21-the-chuckles-2/", "url": "https://benfarrell.com/blog/2008-05-21-the-chuckles-2/", "title": "The Chuckles!", "content_html": "Still waiting on the final video, but this is part of my March Madness of projects. My year has been insane so far with way to much work. Like an idiot, when my friend Jason Burke asked if I could help out with a series of 30 or so illustrations to be finished in less than a month, I said yes!
\nBut, it was fun. I hadn't done illustration work in a while, and had just discovered the gradient mesh tool in Illustrator. The tool allowed me to make some cartoonish, yet 3D like drawings. Yes, I'm a perfectionist, so some didn't come out as good as I'd like, but I'm happy with it overall.
\nFirst, I'll show the one that creeped out my wife the most. This is the forced child clothing labor camp:
On the next drawing I learned to use the pattern brush in Illustrator a little better than my previous attempts at it. It was incredibly useful for painting the leaves on in various configurations, and the grass as well. And there's tons more - but lots were small, here's the rest I put in my portfolio:
\n
Thats it! I haven't blogged since February, but I do have more projects coming down the line - they just need to launch. I have worked hard this year, it'll be nice when I have something to show for it.
\n", "date_published": "2008-05-21T00:00:00Z" } , { "id": "https://benfarrell.com/blog/2008-05-21-budget-hero/", "url": "https://benfarrell.com/blog/2008-05-21-budget-hero/", "title": "Budget Hero", "content_html": "We just launched a new application at 360KID! I wish I could say I did all of it, or even most of it, but I did step in to help a lot during the final stages - just so I can say I took all the credit from the project's real programmer Clint Little. I have a few Flex projects under my belt now, and they all look fantastic compared to most of the ones I've seen out there. This one is no exception thanks to the great animation and illustration my company does. I look forward to launching more and more Flex applications that have a fun look and feel. We're so close to launching our other projects, I'd love to show them since I was the main developer - but this one is another one to be proud of regardless of who did it.
\nThis game called Budget Hero was developed for American Public Media, and allows you to play cards to adjust the federal budget and see if you can balance it. Also allowing you to earn badges, you can see if you're also doing your part to maintain the military, health care, etc. It's up to you to make sure the stuff you think is important doesn't wither away and STILL maintain the budget.
\n![\"Budget](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/portfolio/BudgetHero.jpg\")
I 've been working with Flex 3 for a little while now, and started my first commercial Flex project in December. Luckily I got the CS3 Master Suite...whoo! So since it's more fun to draw in Illustrator, that's what I've been doing. In fact, on this latest project I've been doing, I've been having our artists deliver Illustrator files, then I just export SWF asset libraries. I take these files and embed them right into Flex! It's pretty nifty, and I've been loving the workflow.
\nWell, the project is going full steam, and I've got a bug on my plate. I have some scrolling lists full of graphics. These lists are styled with a background image using 9 Slice Scaling. At first glance it's fine, but I've been noticing some weird redraw issues in the project. It's pretty subtle, but the bounding box of these scrolling images will discolor the background image. More specifically, if I have a colored square layered over another colored square as my Illustrator artwork, the bug will take away parts of the square in the forefront revealing whats behind it.
\nI tried so many different export options, and nothing worked! Finally I copied and pasted this same art into Flash CS3 to produce the same SWF asset library, and guess what? No problems.
\nThere must be some buggy SWF producing code in Illustrator CS3. I hope I'm doing something wrong, but I can't imagine what. But I guess I'll use Flash when stuff breaks on me, oh well.....the Illustrator high was fun while it lasted.
\nTo see what I mean, check out this Flex Application. The Flash list on the left scrolls fine, but check out the Illustrator list. If you scroll and look at the bottom, you can see the redraw issue.
\n", "date_published": "2008-01-15T00:00:00Z" } , { "id": "https://benfarrell.com/blog/2008-01-04-its-been-10-years-since-parappa-now-what/", "url": "https://benfarrell.com/blog/2008-01-04-its-been-10-years-since-parappa-now-what/", "title": "It's been 10 years since PaRappa - now what?", "content_html": "In the US at least, it's been a whole decade since PaRappa the Rapper came out for the Playstation One. It's described as one of the first rhythm video games. Japanese creator Masaya Matsuura gave a microphone to a rapping dog, and it was a hit! It inspired a sequel on the Playstation 2, and a spinoff on the Playstation One, where instead of rapping, rockstar UmJammer Lammy strummed a guitar.
\nPa Rappa the Rapper and UmJammer Lammy
\n![\"PaRappa](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2008/01/parappa.jpg\")
\n![\"UmJammer](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2008/01/umjammerlammy.jpg\")
Rhythm games work by making the player, basically, press buttons to a beat. I'm sure you all know this...you've probably all played Dance Dance Revolution or Guitar Hero. In fact it seems that the only innovation to come along since PaRappa has been to trade your thumbs for your legs, or a guitar.
\nDance Dance Revolution was released in the Japanese arcades in 1998, came to American arcades in 2000, and was finally released by Konami for the Playstation in 2001. DDR, as it's affectionately called, has spawned countless sequels and spinoffs featuring new music and characters. Like PaRappa, DDR players pushed buttons in time to the music. Only, instead of using a normal videogame controller, players used a gigantic gamepad and pushed the buttons with their feet by dancing.
\nDance Dance Revolution, Guitar Hero, Karaoke Revolution, Donkey Konga, and Rock Band
\n![\"Dance](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2008/01/ddr.jpg\")
![\"Guitar](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2008/01/guitar_hero_screenshot.jpg\")
![\"Karaoke](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2008/01/karaokerevolution.jpg\")
![\"Donkey](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2008/01/donkeykonga.jpg\")
![\"Rock](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2008/01/rockband.jpg\")
Next in 2003 came Karaoke Revolution. Karaoke does a bit more than judge your rhythm, it also judges the pitch of your voice through an included microphone.
\nKaraoke Revolution was promptly followed in the US by Donkey Konga and Taiko Drum Master somewhere between 2004/2006. These games included drums to beat in time with the game's soundtrack.
\nIn 2005, along came Guitar Hero. Guitar Hero, again lets players push buttons in time to the music, but this time the buttons were on a custom guitar game controller.
\nAnd, just this past December, of course, Rock Band was released for XBox and Playstation. Rock Band combined the guitars, drums, and microphone to let a group of people play a song together.
\nAmidst the sequels and spinoffs, there's been some creative attempts to stand out from the norm. Elite Beat Agents for the Nintendo DS allowed the player to tap out rhythms all over their screen. Bust A Move allowed players to build up their sequences to fight their opponents. Vib Ribbon looked the craziest, and allowed the user to load their own songs in to play with the game.
\nElite Beat Agents and Vib Ribbon
\n![\"Elite](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2008/01/elitebeatagents.jpg\")
![\"Vib](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2008/01/vibribbon.jpg\")
So...in summary:
\nI'm not bashing any of these games. Least of all Guitar Hero or PaRappa. Heck, I even loved PaRappa the Rapper 2, and UmJammer Lammy. Unfortunately, all of these games have the same tired gameplay that nobody has revamped in 10 years, no matter what cool hardware controller you stick in front of it. Button icons slid across the screen. When the button arrives at its destination, you're supposed to hit the corresponding button on whatever controller you have. It's kinda boring when you put it like that.
\nPaRappa added a weird, fun story line, and wacky songs, and it was a hit. DDR, Guitar Hero, and Rock Band added hardware and suddenly you're doing more than playing a video game. What happens when you master all the songs? It's not really fun anymore. I beat PaRappa the Rapper 2 in 30 minutes, and it wasn't fun anymore. Guitar Hero was just ridiculously difficult towards the end, and just got a little frustrating.
\nAll in all though, they were fun. But again, same old tired gameplay. So it's been 10 years since PaRappa. Now what?
\n", "date_published": "2008-01-04T00:00:00Z" } , { "id": "https://benfarrell.com/blog/2007-12-31-brand-new-yellow5labscom-for-2008/", "url": "https://benfarrell.com/blog/2007-12-31-brand-new-yellow5labscom-for-2008/", "title": "Brand New yellow5labs.com for 2008", "content_html": "I've had yellow5labs.com for awhile, probably since 2001 or so. It's always been a portfolio site, so I can direct potential employers to view my work. Though, I've recently decided to change the focus of the site.
\nYou see, I've been working at the same place for four and a half years now, which is quite insane since the longest job I ever had was a year. It's just my bad luck I guess, and the nature of the career I'm in. The places I work at keep going under (and this includes the grocery store in Spencer, MA that I worked at in junior high school).
\nAnyway, I'm really not planning on quitting or getting fired anytime soon, soooooo I don't really have much need to find a new job. I'll still keep my portfolio and resume current because you really never know, but as is, there really wasn't much point of me keeping a website going.
\nSo, with no need for a new job, I realize that I am doing a lot of cool things at my job and in my free time. I also realized that my old site didn't lend itself to being updated very often with new goings ons. So I'm switching my front page to a blog. The blog will be half personal stuff and half professional stuff. The personal stuff, well...you know, all that stuff my family and friends wanna see - like new photos and life in general. The professional stuff will be a mix of technical articles, projects that were finished at work, and side-projects I'm working on now.
\nAs an expert in Adobe software products, and as the Raleigh Durham Adobe User Group manager, I hope to offer some informative posts. Anyway it'll be a mix.
\nMy interests for 2008 are shaping up to be a mix of web comic creation, photo managing software in Flex/AIR, interactive music experimentation, and a social networking type idea. More or less on these as they do or don't pan out (translation: if I do or don't get bored of the ideas in my head).
\nAnyway, that's where I'm taking this site. I'm sorry to be so trendy and have a blog, but that's what happened. I also happen to have a decent Photo album here. I'll be adding lots more photos soon, and I'll make my wife do the same. I'll also be organizing my portfolio in the same way so that I can ditch the "Coming Soon" page. But that's a lot of organization - I just finished the fun work of learning XHTML/CSS design for the website, and Flex 3/AIR for my photo album management. The rest of the work will just be boring stuff, so hopefully I don't slack off.
\nI expect to have some posts in January on some cool stuff I'm working on, as well as some photos of me and my wife's trip to Vegas over the Holiday.
\n", "date_published": "2007-12-31T00:00:00Z" } , { "id": "https://benfarrell.com/blog/2007-07-01-fantasy-novel-map/", "url": "https://benfarrell.com/blog/2007-07-01-fantasy-novel-map/", "title": "Fantasy Novel Map", "content_html": "My lovely wife is working on a fantasy novel as we speak. A while ago she asked for me to draw a map to help her visualize her world as she wrote.
\n![\"land\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2011/08/land.jpg\")
This was a collection of 3 games we made for NickJr's MyNoggin website.\nI do remember having to do some pathfinding for the Blue's Clues maze, which was a bit of a challenge.
\n![\"tictactoe\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/portfolio/MyNoggin+-+Tic+Tac+Toe.jpg\")
\nGo Diego Go Tic Tac Toe
![\"maze\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/portfolio/blues-cluesmaze.png\")
\nBlues Clues Maze
![\"wildwest\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/portfolio/backyardigans-WWRoundup.png\")
\nBackyardigans Wild West Roundup
Backyardigans: Mission to Mars was a CD-ROM game we created at 360KID. It featured several games for this Nickelodeon show that were strung together by cut-scenes to create a great story where the Backyardigans journey to Mars.
\nThe game was comprised of several Flash modules (cut-scenes and games). Multidmedia's Zinc was used to package this as a desktop application.
\n![\"mtm\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2011/12/mtm.jpg\")
\nAvoiding falling meteors on Mars
When Amazon offers 30 second samples of music to buy, can you make a game out of it!\nYes! I had an idea for a "Name that Tune!" party game where you could use Amazon's API and\nentire music catalog in a desktop built Flash application.
\nThis was just a fun experiment I made myself, and we played it at a get together I had with friends.
\n![\"cosmo_bubbles\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/portfolio/NameThatTune.jpg\")
\nShared party gameplay screen
I think I mainly just architected the framework and developed the main menu container app for this project. Another one that I'm documenting now and can't remember much\nall these years later. Here's a snippet from the 360KID (the agency I worked for) portfolio website about this project:
\n![\"pokemon\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/portfolio/pokemon.jpg\")
\nPokemon Learning League
The Pokemon Learning League is a web-based series of standards-driven, interactive, and animated lessons for children in grades three through six. Working off of initial scripts, 360KID was responsible for storyboarding, animating, and programming 140 different lessons. Each lesson consists of several minutes of animation that perfectly blends educational content with one of the world's most recognized brands, explaining each concept in the style of a mini episode, followed by two interactive sequences where children can be tested on their comprehension.
\nPokemon Learning League went on to receive wide recognition, including:
\n![\"pokemon\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/portfolio/pokemon.png\")
\nPokemon Learning League Dashboard
Usually doing Flash work in these days, this was the rare website I did all in HTML/CSS/JS.\nAs it was 2006, it was all table based layouts, and with the weird non-rectangular shapes, it was a lot of\ndetailed work to get it looking right.
\nThe site was for kids of military families to help them understand and cope with the deployment of a parent.
\n![\"sesame](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/portfolio/sesame-military.png\")
\nListen, Talk, Connect Website
Playground Discovery by AT KidSystems featured a loveable robot named Cosmo. Cosmo led kids around through various activities as they popped bubbles, flew around in a jetpack, and jumped on springboards.
\n![\"cosmo_bubbles\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/portfolio/cosmo_bubbles.gif\")
\npopping bubbles with Cosmo
![\"cosmo_fly\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/portfolio/cosmo_fly.gif\")
\nflying around in a jetpack with Cosmo
The action was driven via a hardware device they called "Mission Control" which featured 4 large pressure sensitive buttons, and a microphone used for volume detection to encourage kids to yell and cause an action to happen.
\n![\"learningsystems\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/portfolio/learningsystems.jpg\")
AT KidSystems wrote a Director Xtra that we at 360KID used to take advantage of the hardware piece. As was common to do at the time, the game was written in Flash, and wrapped in a Macromedia Director shell. The shell handled writing to databases for gameplay analytics, hardware control, login/logout, and loading the Flash based game modules.
\nThe game was geared towards kids with learning disabilities and for parents to monitor the kids gameplay.
\n", "date_published": "2006-01-01T00:00:00Z" } , { "id": "https://benfarrell.com/blog/2005-12-26-sesame-street-interactive-cable-television/", "url": "https://benfarrell.com/blog/2005-12-26-sesame-street-interactive-cable-television/", "title": "Sesame Street: Interactive Cable Television", "content_html": "This was quite the interesting project. I was only a little involved the first year, and the second year this project resurfaced I created a new game. And finally the 3rd year this project resurfaced, I managed the refresh and the new game that we added.
\nThe reason that it was refreshed every year was to finish up in time for the Emmy's. This project was designed to run on a user's cable set-top box, and as such won an Emmy nomination each of the three years. We unfortunately lost the category as our client Sesame Street couldn't match the big budget project that winners like NFL could put together. Nevertheless it's the project that makes me an Emmy nominee!
\nIt worked by serving up Flash projects via a technology provided by Blue Streak out of Montreal. It worked on newer cable set top boxes. It was a GREAT experience tailoring our Flash games to remote control input and extremely low processor and memory restrictions for our heavily animated and voiced-over games.
\n![\"ictv\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2011/12/ictv.jpg\")
\nCounting Eggs with Chicken Dance Elmo
Here's another project I worked on, but almost 20 years later I don't remember much about. Here's a snippet from\nthe 360KID portfolio website about this project:
\nLike many children, Cactus Cal is having trouble staying dry at night. To keep him from flooding the desert during the night, his friends assist him in creating a personalized Dry as a Desert Plan. 360KID created the characters, storyline, user interface, and carnival games for the Dry2Night program. It was designed to meet the needs of both parents and child to help them to understand the physical and emotional causes of bedwetting. The website contains nine Flash games, holds a large back-end database, and allows for a personal and unique experience for each user, parent or child.
\n![\"dry2night\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/portfolio/dry2night.jpg\")
\nDry2Night
Back when I lived in Boston....lets say 2001 to 2005, I was in a rock band called Mercury Charm Offensive. I played keyboard, and well I was their resident graphic and web designer. I designed our logo too, see below!
\nOur first album was called "Fancy Plans and Pants to Match", named after a Jimmy James NewsRadio quote.
\n![\"fancyplanscover\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2011/08/faancyplanscover.jpg\")
Here's the back cover:
\n![\"back](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2011/08/back-cover-sized.jpg\")
And then our second album (I stole the photo from my wife):
\n![\"cover\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2011/08/cover.jpg\")
And as I said...I did the web design
\n![\"Mercury](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2011/08/Mercury-Charm-Offensive-Web-Site.jpg\")
![\"about\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2011/08/about.jpg\")
There was also miscellaneous graphics for buttons, show flyers, etc
\n![\"mco\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2011/08/mco.jpg\")
![\"mcAOK\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2011/08/mcAOK.jpg\")
![\"sillyfaces\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2011/08/sillyfaces.jpg\")
This was a quick one, but kind of unique. Basically this was simple Flash embed with a button to print out\na set of activity pages for this Nick Jr. show.
\n![\"lazytown\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/portfolio/Lazytown.jpg\")
\nPrintables from a Lazytown Flash embed
We did this grant based project for a company called Inflexxion through my then employer 360KID. The project was an interactive story that took kids through several playable missions to help them deal with all the emotions and troubles that come with having a divorce in their family.
\nThere were many walkthroughs and interactive tutorials, a journal they could write in, and more.
\nThe project was done with Flash and divided into many modules as the user explored and went through missions.
\n![\"divorce\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/portfolio/divorce-small.png\")
\nMission Control with Hank the Superdude
![\"divorce\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/portfolio/Inflexxion+-+Divorce.jpg\")
\nIntro animation
This MAY have been my first Actionscript 3 project if I recall correctly. I remember upgrading from AS2 to AS3. One of our devs at 360KID started this internal project, and I did the language upgrade...adding new features along the way.
\nBasically, the idea was to make this a product we could sell and reskin as part of our client services. It was pretty much a kid friendly Photoshop. I added some cool features as new Flash features came along like saving the photo you made to your desktop from the web.
\n![\"kidpaint\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2011/12/kidpaint.jpg\")
\nPainting with 360KID's KidPaint
Awhile back, I had an idea for a comic strip, which turned into an animation, which turned into too big of a project as I went crazy on it as I wrote scripts, turned the characters into 3D models, etc.
\nThe idea was Mr. Bland. Basically an anti-hero but with heroic ancestry who try to get this boring guy to live up to his bloodline.
\nHere is the main character with his titular line "Danger is his mother's maiden name"![\"mr.-bland-sketch\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2011/08/mr.-bland-sketch.jpg\")
Here's a group shot of all the main characters:![\"mr.-bland-characters_final\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2011/08/mr.-bland-characters_final.jpg\")
Here's my original pencil sketches that I scanned in. I later traced the images and converted them to vector through Adobe Illustrator.
\n![\"mr.](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2011/08/mr.-bland-sketch1.jpg\")
![\"DecDanger-sketch\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2011/08/DecDanger-sketch.jpg\")
![\"gwen](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2011/08/gwen-sketch.jpg\")
![\"harland\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2011/08/harland1.jpg\")
![\"juliet)\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2011/08/juliet.jpg\")
![\"drhammond\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2011/08/drhammond.jpg\")
Someday, I'd love to bring this back as a SIMPLE comic strip/graphic novel so I can actually finish a storyline!
\n", "date_published": "2004-08-04T00:00:00Z" } , { "id": "https://benfarrell.com/blog/2004-02-01-picturetwister/", "url": "https://benfarrell.com/blog/2004-02-01-picturetwister/", "title": "Picture Twister", "content_html": "I don't think this was tied to a particular client, but it was a fun little proof of concept.\nRemember, this was around 2004 - so mobile was pretty much non-existent. Yet, we made a mobile game for the Pocket PC!\nIt was made in Flash then dropped on the device to work with the stylus. Its a simple game where you move the pieces of a scrambled picture\ninto place.
\n![\"picture](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/portfolio/picturetwister.png\")
\nPicture Twister
We at 360KID did a few games for PBS Kids with Sagwa. We did two similar style games. Counting Koi was a simple counting games where kids would count fish in a pond, and Exploring the Palace was mostly a click and explore type of game.
\nThe logic and gameplay in these games aren't incredibly difficult - though weaving animation/dialog clips together and fitting a particular narrative for the infinite number of possibilities can be hard. We did get it down to a science at 360KID for these click and explore games with characters that guide you through the experience.
\n![\"countingkoi\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2011/12/countingkoi.jpg\")
\nCounting Koi
![\"exploringthepalace\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2011/12/exploringthepalace.jpg\")
\nExploring the Palace Game
My first "mobile" project! This was before Android, before iOS....
\nIt was on the Pocket PCs back in the early 2000's. We were doing a stylus based Flash project for Highlights magazine on the smartphone-like pocket organizer. We ran Flash with a wrapper program called "Ant" which was designed to run on Windows CE.
\nThe mobile app was designed to be the interactive version of Highlights Hidden Pictures, where the illustration hid several objects that the user was supposed to find.
\n![\"highlights\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2011/12/highlights.jpg\")
\nHighlights Hidden Pictures - Found a boomerang and a shoe
I want to say this was one of my first projects at 360KID (a creative agency focused on children's educational media).\nIt was a simply interactive storybook for PBSKids featuring Mr. McFeely from Mister Rogers' Neighborhood.
\n![\"speedydelivery\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/portfolio/mrrogers.png\")
\nSpeedy Delivery Interactive Storybook
Cats Incredible Journey was a game created while I was at 360KID. It focused on helping kids with emotions if I recall correctly. In it's prototype phase, it was a Flash game with limited interaction and awesomely designed animated sequences.
\nThe reason behind the interaction and gameplay being so limited and lightweight was so we could port it to a straight DVD which could be played on your standard set top player using the arrow keys and menu buttons to initiate actions.
\nThe Flash sequences were rendered to Quicktime movies which I brought into DVD Studio Pro. It was a pretty difficult to pull of, as I think I remember only being able to work with 6 variables throughout the entire application.
\nWas a frustrating thing to produce a DVD game, but we pulled it off!
\n![\"catsincrediblejourney\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2011/12/catsincrediblejourney.jpg\")
\nCat greeting us as he goes to collect jewels in the woods
While at 360KID, we created a interactive CD-ROM for the Discovery Channel. The aim was to create a series of modules for kids to explore various scientific topics.
\nI didn't have a huge part in this project - it was mostly maintenance as I came in at the end, but I did develop one of the 20 interactive modules - which was a cost/benefit activity for heating and cooling a home.
\n![\"discoverychannel\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2011/12/discoverychannel.jpg\")
\nSolar Energy Activity
This was a Shockwave Director based interactive learning experience. I created the interactivity in this and stictched together the various parts.
\n![\"mcgrawhill\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/portfolio/McGraw+Hill+-+In+Psych.jpg\")
\nMcGraw Hill Textbook supplemental
This was a marketing CD-ROM for a Phillips product called Intellivue - a networked system of patient monitoring systems. I did the Director engineering work, and integrated the user interface with Flash assets produced by the artists. There were several pre-rendered rotating 3D products, and several high quality videos.
\n![\"intellivue\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/portfolio/philips.jpg\")
\nPhilips Intellivue Patient Monitor
![\"intellivue\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/portfolio/Philips+Medical+-+Sales+Demo.jpg\")
\n(faux, pre-rendered) 3D rotating view
This was a surgical simulation tool which was animated and designed in Flash. I took the Flash assets and brought them into Director. From there it was broken up into little interactive steps to complete the surgery.
\n![\"surgery\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/portfolio/Smith+and+Nephew+-+Surgical+Simulation.jpg\")
\nSmith & Nephew Surgical Simulation Tool
This educational CD-ROM was shipped with the Sabias Que text book as a supplement to teach the Spanish Language. I engineered the entire project, and used XML data to make up the hundreds of questions, which required me to build a rendering engine to render these question from the XML. Our navigation system was made in Flash and integrated with Director.
\n![\"sabiasque\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/portfolio/McGraw+Hill+-+Sabias+Que.jpg\")
\nSpanish language textbook companion
Along with doing the HTML for the website, I created a content management system in Director. The system used FTP to read and write files from the website, so the client could go in and edit the site without transferring files or coding HTML.
\n![\"makeamovie\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/portfolio/Thomas+Gregory+Associates+-+Site+Editor.jpg\")
\nWebsite CMS desktop application
There were quite a few of these games, and I did quite a few along with a couple other developers. This one in particular, I used some calculus to push clouds along vector lines to produce random but predictable weather patterns.
\n![\"teachersuniverse\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/portfolio/Teachers+Universe.jpg\")
\nWeather prediction simulation
My first official job after school was at a awesome company called Juniornet. Think America Online\nbut for kids! We had all sorts of activities for kids even including kids email.
\nThis project I worked on was allowing kids to make and share movies, by dragging and dropping\npre-made assets on their stage, selecting a background and recording the assets moving.
\n![\"makeamovie\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/portfolio/Juniornet+-+Make+a+Movie.jpg\")
\nJuniornet Make a Movie
Back in college, I took a fair number of 3D animation classes. We used a tool called Softimage, but if I remember right, I started out with 3D Studio MAX. Anyway here's a couple 3D renderings from back in the day
\nFirst off, some friends and I had this grand idea for an animation revolving around a mad scientist's lab. We never finished (and actually barely started if I remember right. But I did a few lab equipment renderings.
\n![\"\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2011/08/Chemistry-Set-3D-Scene.jpg\" "\\"Chemistry")
![\"\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2011/08/Science-Lab-3D-Scene.jpg\" "\\"Science")
Next, here's my first 3D scene I ever did. It was a tutorial out of a 3D Studio MAX book:
\n![\"\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2011/08/Dinner-3D-Scene.jpg\" "\\"Dinner")
Finally, lets talk senior thesis! I made up an entire legend of the Cancerbot 9000 project. Since the cigarette companies couldn't advertise to kids, they manufactured a fleet of robots to do it instead. And the fake advertising propaganda ensues!
\n![\"\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2011/08/Cancerbot-9000.jpg\" "\\"Cancerbot")
I actually forgot what the deal was here...but I was doing some Grocery store themed 3D animation that never panned out. Here's a still....but I do remember the final video being kinda trippy.
\n![\"\"](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2011/08/GROCERY.jpg\" "\\"GROCERY\\"")
Another embarrassing entry in my design portfolio! I think this was the first paid website I ever did. It looks pretty bad to me now, but it was done in like 1997. The company was pretty happy with what I did, but the funny story was that I was learning Macromedia Dreamweaver at the time. They wanted it done with Adobe GoLive. I knew at the time that HTML files were HTML files, and any IDE would do the trick.
\nI forgot the GoLive request months later as I continued on in Dreamweaver, and wrapped up the site as a Dreamweaver site file and passed it off.
\nI think they were kinda mad - thinking that the project was a goner. I spent the next few minutes importing the site over while they eyed me suspiciously. My bad! They were nice folks despite our misunderstanding.
\n![\"Simmetrix](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/2011/08/Simmetrix-Web-Page.jpg\")
Back in college, I had a part time job at the "Academy of Electronic Media" which was a university organization.\nI did various multimedia projects with them, and this was a Director based application where you could enter algebraic function\nand it would graph them for you, as well as offer helpers to understand what was happening.
\n![\"Hidden](\"https://d2ypg8o05lff0b.cloudfront.net/wp-content/uploads/portfolio/Hidden+Curves-Academy+Electronic+Media.jpg\")