Skip to content

Adobe AIR, NativeProcess and Python – Introduction

Right now, if you want to handle input and output between Arduino and Adobe Flash/AIR, the recommended best practice is to use a program like SerProxy to listen to your serial input, turn it into socket data and publish it to local host. Inside of Flash, then, you treat it as a socket server – you listen for socket data, and respond in kind.

It works, but it’s not pretty. You need to have access to the right ports, and you have to run a third-party application with its own config file to make it work. Additionally, I’ve never once been able to get it working on a mac – I know I’m not the only one, either. There’s some weird shit going on there.

I decided to see if it was possible to get the same results using the new NativeProcess API in Adobe AIR 2. Read on for the details!

Overview – Selecting a Language

So, before we get started let’s define the problem as specifically as possible. We have a device, in this case an Arduino Microcontroller, that’s reading and writing to a serial port on the computer. Our challenge is to allow our Flash application, running in AIR, to interact with it. Flash, however, has no direct access to serial port data – even with the expanded AIR runtime that stuff is still sandboxed away. So we know right out of the box that we’re going to need an additional language to handle transport between the serial port and AIR.

My challenge is this: I’m a flash guy. I can comfortably scan other languages, but I don’t know the Java or Python or C API’s or syntaxes as well as I probably should. So I need to pick a language that’ll make this as easy as possible to implement. I first looked at the source of SerProxy to see if I could repurpose the existing code there – but it frankly scared me a little bit. C can be an intimidating thing to look at, and when there are 15 files full of one-or-two-character-long method names and system calls I have to assume that there’s going to be an easier way. The advantage is, there are no dependencies – once you have your program written, you can compile it down to an executable on any system, and that’s pretty neat. Still, though – too scary for now, I want to be up and running quickly.

So I looked into Java – and that made a bit more sense, but there seem to be issues with the libraries required for native serial IO and the documentation in the tutorials was a bit long. I wanted something simple. Bookmark the Java pages for future reference, but keep looking.

So I looked into python, and apparently serial IO can be handled with about two lines of code, both legible and logical. The downside is that the user needs to have Python installed, and in addition needs to have the PySerial library installed. This is a minor pain in the ass, but it’s not that big a deal – anyone who is going to be using my code should be able to at least install python and pyserial, fine. Plus, Python runs on any operating system as long as you have the python interpreter installed. So, Python it is.

Setup

Really simple. Open your AIR IDE of choice, create a new Flex project. My MXML is below, it’s all one file – paste that in there. Then right click the src folder and add a new file, call it “ThreadedSerialInterface.py” – the code that will go in here is below, as well. That’s it. The nice thing about AIR is that it will actually bundle your script in with your AIR application, so there’s no need to worry about running multiple programs the way you have to now, with SerProxy.

The Python Script

I know that I want to be able to write a script which is going to remain open in the background until I tell it to close. While open, this script needs to be able to do two things – accept any data coming out of the serial port and print it to stdout, and accept any data coming in from stdin and write it out to the serial port. Simple enough, but these are two distinct processes – so I googled around a bit and decided to implement these processes as two threads within my python script. Here’s the script as I currently have it, with the caveat that I am NOT a python developer and all of this is either made up as I went along or cribbed from helpful folks on Stack Overflow. So, here’s my current script:

from sys import argv, stdin
from serial import Serial
import threading
 
try:
    baudRate = int(argv[-1])
    serialPort = argv[1:-1]
except ValueError:
    #some nice default
    baudRate = 9600
    serialPort = argv[1:]
 
#support paths with spaces. Windows maybe?
serialPort = ' '.join(serialPort)
if not serialPort:
    exit("Please specify a serial port")
 
print( "opening connection to %r@%i ..." % (serialPort, baudRate) )
ser = Serial(serialPort, baudRate, timeout=1)    
 
 
 
class readerThread(threading.Thread):
    def __init__(self, threadID, name):
        threading.Thread.__init__(self)
        self.threadID = threadID
        self.name = name
 
    def run(self):
       while True:
           if ser.inWaiting() > 0:
               print( "[R]" + ser.readline() )
 
 
 
class writerThread(threading.Thread): 
    def __init__(self, threadID, name):
        threading.Thread.__init__(self)
        self.threadID = threadID
        self.name = name
    def run(self):
        while True:
            input = raw_input("[W}Give Me Input!")
            print ("[W]You input %s\n" % input.strip())
            ser.write(input.strip())
 
 
thread1 = readerThread(1, "readerThread")
thread2 = writerThread(2, "writerThread")
 
thread1.start()
thread2.start()

A few notes: I’ve appended “[R]” and “[W]” to outputs from my Reader and Writer threads, respectively. This lets me easily separate them in my AIR app, which is helpful for knowing what I’m looking at.

So what’s going on in this script? I instantiate an instance of pySerial’s serial class, called “ser”, and I tell it to connect to the serial port and baud rate specified as arguments when I run this script. Simple enough. Then I define two classes, readerThread and writerThread, each of which extends threading.Thread. readerThread simply loops and, if there is any output from the serial port, it prints it out. writerThread is the inverse – it loops, and waits for input from stdin each time through. It takes that input, when it gets it, and writes it to the serial port. I start both threads and I’m good to go.

Let’s see how we can tie this into Flash.

AIR and NativeProcess

This is actually a lot more painless than you might expect, once you understand what’s going on. AIR 2 has exposed the NativeProcess API, meaning that we basically have access to a command line interface directly through our AIR application. Rather than stepping through every detail, I’ll just post my code below, with comments. Right now it’s all one file:

<?xml version="1.0" encoding="utf-8"?>
<s:WindowedApplication xmlns:fx="http://ns.adobe.com/mxml/2009" 
					   xmlns:s="library://ns.adobe.com/flex/spark" 
					   xmlns:mx="library://ns.adobe.com/flex/mx"
					   closing="killProc()">
	<fx:Declarations>
		<!-- Place non-visual elements (e.g., services, value objects) here -->
	</fx:Declarations>
 
	<fx:Script>
		<![CDATA[
 
			// create our native process
			private var np:NativeProcess = new NativeProcess();
 
			// create our native process startup info, which will hold things like the path to python and any arguments.
			private var npi:NativeProcessStartupInfo = new NativeProcessStartupInfo();
 
			/**
			 * 	This method gets called when the user clicks the "init" button. It basically makes sure that your python 
			 *  path is correct, then assembles the NativeProcess call, assigns listeners and starts the process.
			 */
			private function setup():void {
 
				// the file we're actually going to be executing is python itself - the rest, my script included, gets passed in as arguments.
				var exec:File = File.applicationDirectory.resolvePath(pythonLocation.text);
 
				// check to see if the python path resolved correctly.
				if (exec.exists) {
					initButton.enabled = false;
					killButton.enabled = true;
					npi.executable = exec; // assign "exec" to be the executable in our startup info.
					var args:Vector.<String> = new Vector.<String>; // create a vector to hold arguments - don't assign til the end.
					args[0] = "-i"; // python needs this to handle I/O correctly, details to follow.
					args[1] = File.applicationDirectory.resolvePath("ThreadedSerialInterface.py").nativePath; // this is the path to our python script, included with the AIR app.
					args[2] = serialPort.text; // this is the serial port, "COM3" etc on pc or "/dev/tty.usbserial-XXXXXX" on mac, etc.
					args[3] = baudRate.text; // this is our baud rate.
 
					npi.arguments = args; // assign the arguments to the startup info object.
 
 
					// add listeners for stdout, stderr and process exit.
					np.addEventListener(ProgressEvent.STANDARD_OUTPUT_DATA, stdoutHandler);
					np.addEventListener(ProgressEvent.STANDARD_ERROR_DATA, sterrHandler);
					np.addEventListener(NativeProcessExitEvent.EXIT, exitHandler);
 
					// start the NativeProcess
					np.start(npi);
				}
			}
 
			/**
			 * 	this method passes a string to our native process, which passes it to the microcontroller.
			 */
			private function writeToSerial():void {
				np.standardInput.writeMultiByte(toSerial.text + "\n", File.systemCharset);
				toSerial.text = "";
			}
 
			/**
			 * 	This method handles output from the NativeProcess. I have some basic parsing logic in here to separate output by thread.
			 */
			private function stdoutHandler(e:ProgressEvent):void {
				var output:String = np.standardOutput.readMultiByte(np.standardOutput.bytesAvailable, File.systemCharset);
				if (output.indexOf("[R]") > -1) {
					logOutputFromReaderThread(output.slice(3));
				}
 
				if (output.indexOf("[W]") > -1) {
					logOutputFromWriterThread(output.slice(3));
				}
 
 
				trace("[Output from Python] " + output);
 
			}
 
			/**
			 * 	utility function to log Reader thread output.
			 */
			private function logOutputFromReaderThread(s:String):void {
				var text:String = s += readerOut.text;
				readerOut.text = text;
			}
 
 
			/**
			 * 	utility function to log Writer thread output.
			 */
			private function logOutputFromWriterThread(s:String):void {
				var text:String = s += writerOut.text;
				writerOut.text = text;
			}
 
			/**
			 * 	This handles STDERR output, right now it's just tracing.
			 */
			private function sterrHandler(e:ProgressEvent):void {
				var output:String = np.standardError.readMultiByte(np.standardError.bytesAvailable, File.systemCharset);
				trace("[Error!] " + output);
			}
 
			/**
			 * 	This fires when the native process is terminated. 
			 */
			private function exitHandler(e:NativeProcessExitEvent):void {
				trace("Native Process has terminated!");	
				initButton.enabled = true;
				killButton.enabled = false;
			}
 
			/**
			 * 	This forces the NativeProcess to close. You must call this when you quit your program, 
			 *  or else your processes will live on.
			 * 
			 *  Notice that the Application is using this to handle the Closing event - so when you close the app,
			 *  your process is killed.
			 */
			private function killProc():void {
				trace("Killing native process");
				np.exit(true);	
			}
 
		]]>
	</fx:Script>
 
	<!-- MXML for layout, self-explanatory I hope -->
	<s:TextInput x="105" y="16" width="174" id="pythonLocation" text="C:\\Python26\\python.exe"/>
	<s:Label x="10" y="16" text="Python Location" height="22" verticalAlign="middle"/>
	<s:TextInput x="105" y="50" width="174" id="serialPort" text="COM3"/>
	<s:Label x="10" y="50" text="Serial Port" height="22" verticalAlign="middle"/>
	<s:TextInput x="105" y="80" width="174" id="baudRate" text="9600"/>
	<s:Label x="10" y="80" text="Baud Rate" height="22" verticalAlign="middle"/>
	<s:Button id="initButton" x="10" y="110" label="Init" width="269" click="setup()"/>
	<s:Button id="killButton" x="9" y="143" label="Kill Process" width="269" click="killProc()" enabled="false"/>
 
	<mx:HRule x="10" y="172" width="268"/>
	<s:Label x="11" y="189" text="Write to Serial:" height="20" verticalAlign="middle"/>
	<s:TextInput x="98" y="187" width="174" id="toSerial"/>
	<s:Button x="10" y="217" label="Send" width="262" enabled="{toSerial.text.length}" click="writeToSerial()"/>
 
	<mx:HRule x="10" y="251" width="268"/>
	<s:TextArea x="10" y="292" width="435" height="299" id="readerOut"/>
	<s:TextArea x="453" y="292" width="421" height="299" id="writerOut"/>
	<s:Label x="10" y="272" text="Output from Reader Thread:"/>
	<s:Label x="453" y="272" text="Output from Writer Thread:"/>
 
</s:WindowedApplication>

Results

So, this ALMOST works. That is to say, it does work but it’s not quite ideal. I’m having some issues with input and output into python, and I think they’re caused by the fact that I don’t have an actual terminal window. When python is waiting for raw_input(), it outputs a prompt (“>>>”). When I print serial anything out, it’s expecting a console to be able to print to. Because we’re running this as a process from inside of AIR, there is no actual terminal for interface – which brings us to the “-i” argument I’m passing in above.

Now, I’m not a python developer – I’ve made that clear, right? – but from what I can tell, “-i” is telling Python to act as through there’s a terminal even though there isn’t. It’s saying “Trust me, someone is listening even if you can’t see them.” Without that argument, the script waits until it exits to output anything. Obviously that’s no good for our purposes because we want this thing to be constantly running, listening for output. So, the “-i” argument helps us out a lot – but there’s one final facet of this puzzle that’s throwing me for a loop.

When I run this application, it works great – except that only every other command I try to write to the serial port goes through. Boot it up, run it and see. Hit the init button, wait for the “Initialized, waiting for input.” response from the Reader thread, type a number into your “Write to Serial” box and hit submit. The Writer thread should respond with “You input 1″, and if your arduino unit is doing anything in response to serial input it should happen. Now type another number and hit submit – nothing happens. Type a third and hit submit, everything is fine.

I was confused until I ran it in debugger mode, and realized that I’m getting output from STDERR – specifically, every other time I submit serial input it returns “>>>”, a prompt, via the Error stream. I have no idea why.

See comment thread for more information – this problem was solved! Specifically, I replaced the “-i” command line option with a “-u”.

Conclusion

This is obviously a work in progress, and I’m going to update this as I go. I’m writing this post today, perhaps a bit prematurely, because there’s been some noise on Twitter lately about other folks trying to do similar stuff and I thought I’d throw this out there in case it helps anyone. If you have any feedback, I’d love it in the comments below – especially any feedback on the python section. I’m not 100% understanding every part of it.

I think as Flash developers many of us tend to live in very small boxes (albeit boxes with lots of cool stuff inside). This project has helped me to realize just how little I understand about programming in general, and in that regard it’s been very helpful. I hope to continue learning as I go. :)

{ 10 } Comments

  1. reelfernandes | November 9, 2010 at 8:21 am | Permalink

    I also don’t know Python syntax or API at all. Maybe the API docs will explain the -i argument ( possibly a verbose arg ) and give you more insight into it.

  2. Phillip Senn | November 17, 2010 at 8:43 pm | Permalink

    Thanks for posting this!
    I’m a ColdFusion developer looking to access the serial port for a bar code scanner.

  3. mykola | November 17, 2010 at 8:45 pm | Permalink

    Hey Phillip,

    Cool. I haven’t looked into CF at all, but it seems like most of the server-side language get native access to serial data. That might be your best bet to start – if not, Python with PySerial is certainly easy. I still haven’t managed to work out that one little kink, though. No time! If you get something cool up and running, please do link back!

  4. fangzx | November 23, 2010 at 3:21 pm | Permalink

    This posting is very helpful. I also see the ‘getting output from STDERR ‘ problem, hope this problem will solved soon.

  5. Will | January 3, 2011 at 1:12 pm | Permalink

    Nice work Mykola,

    I’m wondering how well the Threading works in your Python script. The reason I ask is because in my own experience with Threads I find that using raw_input in one thread will actually block the print in other threads. I believe raw_input blocks stdout until something is entered. Despite the threads running independent of each other they are still sharing stdin, stdout.

    If you’re interested to see what I mean – http://pastebin.com/YEDnk8Pv

  6. mykola | January 3, 2011 at 6:44 pm | Permalink

    @Will interesting, thanks – but I don’t think I’m having issues with that? My STDOUT seems to be working just fine from the arduino unit. I’m getting STDOUT constantly and it’s got what I’d expect. My issue comes from when I try to write to the Python Script – if you download this and give it a try, you’ll see what I mean. I have to submit every input to STDIN twice, because every other time I submit it seems to crap out.

    Still, you might be right – I’m by no means a Python coder. I’ll look into it more!

  7. mykola | January 3, 2011 at 7:39 pm | Permalink

    Awesome! After reading @WillDaddy’s tweets at me I decided to try swapping out my “-i” option for a “-u” option instead, and that seems to have done the trick. I’m no longer getting that STDERR stutter breaking my attempts to write TO serial!

    Also, when I run this, the input does NOT block the output. Go ahead and just try to manually run this in the command line – make sure your serial port is reading in from an Arduino or something and that you have a steady stream of data coming in. It works just fine, though it looks ugly on the command line since your input prompt gets upscrolled almost immediately and when you input it spreads it across various lines until you hit enter – but it WORKS, and the nice thing is if you’re using an AIR front end like I’m building here then the ugliness gets abstracted out for you.

    Now to do some speed tests, comparing this to a socket! :)

  8. Will | January 3, 2011 at 9:03 pm | Permalink

    Glad the -u is working for you. I like the idea of having a more direct line to serial rather than using a seperate proxy program when running from Flash.

    Oh, and it’s @willdady (one d) :P

  9. mykola | January 3, 2011 at 11:38 pm | Permalink

    Will be doing a new post soon on this subject. My immediate question is, is there any way to use this to transfer image or video data? It would appear to break down, unless I’m actually able to just spit out encoded binary data right to STDOUT? Seems unlikely…

    But, one thing at a time! :)

  10. Will | January 4, 2011 at 12:09 am | Permalink

    I assume you can simply do a writeBytes on NativeProcess.standardInput sending a ByteArray (of an image for example). The problem will be with forwarding that onto the Arduino as it has limited memory (2KB?). I wonder if it would be possible for the Arduino to simply ‘forward’ the bytes on over an XBee and have it read back in at another computer?

Post a Comment

Your email is never published nor shared. Required fields are marked *