-
PyWorks 2008 Slides November 15th, 2008
Slides from my PyWorks 2008 talks are now available. This conference was shared with php|works and I enjoyed this format. It was great to see many of my Python and PHP friends at the same event. Thanks to everyone who attended my talks.
Py65: Microcontroller Simulation with Python
This talk introduced the venerable 6502 microprocessor family, building small computer systems with these parts, and then simulating those systems with Py65.
The audience participation was great. We had fun stepping through some small assembly language programs on the simulator. One attendee wrote:
This was fascinating and the speaker was awesomely enthusiastic. The overview of microcontrollers and their significance was enlightening and entertaining. The simulator design presented was fantastically simple and very Pythonic. I can’t wait to see where this project goes.
Thanks and I’m glad you enjoyed it. For updates on the Py65 simulator, please watch my blog and the Py65 project page on Ohloh.
URL Mapping with Routes
We explored the Routes library from the ground up, setting it up and then exploring its options and matching. We worked through many of the examples with live demos on the Python interactive interpreter.
The talk was attended by several Pylons users, who gained a better understanding of how Routes works by seeing it outside the context of any particular web framework.
URL Mapping with Horde/Routes
Bonus Slides! Horde/Routes is a PHP 5 library that is a direct port of Routes. Since there were so many PHP folks at this conference as well, I ported all of the examples in my Routes talk to work with Horde/Routes.
These slides will help you get acquainted with the PHP version. Since the two presentations are otherwise identical, you might also find it an interesting comparison between Python and PHP 5.
-
Py65 0.1: Introducing Py65Mon November 9th, 2008
Py65 0.1, a 6502 microprocessor simulator written in Python, has been released and is available on the Python Package Index (PyPI). You can now easy_install it:
$ easy_install py65
Py65Mon
Since my initial announcement of Py65, there have been many bug fixes and unit tests added. The most noticeable addition is a new machine language monitor. It will be installed automatically and is started with the
py65moncommand:$ py65mon Py65 Monitor <6502: A=00, X=00, Y=00, Flags=20, SP=ff, PC=0000> .
At the prompt, type
helpfor a list of commands orhelp commandfor help on a specific command. The monitor commands are very similar to the excellent VICE Monitor, so VICE users should feel right at home.The biggest difference from VICE is that the
loadcommand requires a load address as the second argument and starts reading the binary data from byte 0. It does not expect byte 0 and 1 of the file to contain a Commodore-style load address. Also, assembling and disassembling from the monitor are not yet implemented but are planned.Hello World
Just like Michal Kowalski’s 6502 Macroassembler & Simulator for Windows, Py65Mon will trap writes to
$E001and echo the bytes toSTDOUT.This is enough to get us to our first “Hello World” program running under Py65. First, we’ll write a short assembly language program to print the message. Save it as
hello.asm.*=$C000 CHAROUT=$E001 HELLO: LDX #$00 LOOP: LDA MESSAGE,X BEQ DONE STA CHAROUT INX JMP LOOP DONE: RTS MESSAGE = * !text "Hello, World!" !byte 0
We then assemble the program into a binary, using Marco Baye’s Acme Cross-Assembler:
src$ acme --format plain --outfile hello.bin hello.asm
The
--format plainswitch instructs Acme not to prepend the Commodore-style load address in the binary. If you’d like to get going quickly, you can also download hello.bin.With the binary ready, we can start the monitor and load it in:
src$ py65mon Py65 Monitor <6502: A=00, X=00, Y=00, Flags=20, SP=ff, PC=0000> .add_label c000 hello <6502: A=00, X=00, Y=00, Flags=20, SP=ff, PC=0000> .load "hello.bin" hello Wrote +29 bytes from $c000 to $c01c
Py65Mon supports symbolic addressing in most commands. The first command,
add_label, defineshelloas a label for address$C000. The second command loads the binary into that address.We can now set the program counter with the
registerscommand, and execute the code up toRTSwith thereturncommand.<6502: A=00, X=00, Y=00, Flags=20, SP=ff, PC=0000> .registers pc=hello <6502: A=00, X=0d, Y=00, Flags=20, SP=ff, PC=c000> .return Hello, World! <6502: A=00, X=0d, Y=00, Flags=20, SP=ff, PC=c00e> .
Now we have run the program, printed “Hello, World!”, and returned to the prompt. We can see the program counter is left at
$C00E.You can also use the
stepcommand to step through the program. Just set the program counter to the start address again ($C000orhello) and repeatedly enterstep. As you are stepping repeatedly, you can simply hit ENTER to repeat the last command.From here you can also explore other commands, e.g.
mem c000:c003to display the memory in that address range. The default radix is hexadecimal. You can also prefix with$for hexadecimal or+for decimal, likemem +49152:+49155.Next Steps
Py65 and its monitor are now complete enough to run most simple 6502 programs, including many from the 6502.org Source Code Repository. The next versions will include more I/O devices and monitor commands, with the goal of running a sophisticated 6502 program like Lee’s Davisons’ Enhanced 6502 BASIC.
-
PHP Temporary Streams October 17th, 2008
It’s been a while since David Sklar called out to let a thousand string concatenations bloom. That discussion produced some entertaining suggestions for putting strings together such as using
preg_replaceand calling out to MySQL withSELECT CONCAT.Here’s an approach that uses filesystem functions. When combined with some lesser-known PHP streams functionality, it has several practical applications.
Opening Files for Reading and Writing
There are several modes for opening a file that will allow you to seek to arbitrary positions in the file and read or write at those positions. One of the most frequently used of these modes is
w+, which thetmpfilefunction uses automatically.We can repeatedly write, then rewind the pointer and read:
$f = tmpfile(); fwrite($f, 'foo'); fwrite($f, 'bar'); rewind($f); $contents = stream_get_contents($f); //=> "foobar" fclose($f);
When writing to the filesystem, the above provides yet another inefficient solution to David’s exercise. Now let’s take it a bit further to see how this can be useful.
In-Memory Streams
PHP 5.1 introduced two new in-memory streams:
php://memoryandphp://temp. Thephp://memorystream operates entirely in memory. Thephp://tempstream operates in memory until it reaches a given size, then transparently switches to the filesystem.We can modify the above example to use the
php://memorystream instead of hitting the filesystem:$f = fopen('php://memory', 'w+'); fwrite($f, 'foo'); fwrite($f, 'bar'); rewind($f); $contents = stream_get_contents($f); //=> "foobar" fclose($f);
Putting a string inside a fast temporary stream can be very useful. For example, we can then attach filters to that stream.
Testing
Temporary streams are also handy for testing. There are some rather elaborate virtual file system libraries out there but many times a stream is all you need.
Zend_Log has a log handler for streams that accepts either a filename or a stream resource. We can configure it with a
php://memorystream for testing:$f = fopen('php://memory', 'w+'); $writer = new Zend_Log_Writer_Stream($f); $logger = new Zend_Log($writer);
Assuming your well-designed application has a convenient injection point for the logger instance, your test can pass it in before your test activates some action which should result in logging:
$logger->crit('critical message worth testing');
When the action has completed, the test can rewind
$fand use it as a test spy.rewind($f); $contents = stream_get_contents($f); $this->assertRegExp('/message worth testing/i', $contents); // PHPUnit
Not surprisingly, my unit tests for Zend_Log use this same technique.
Application Usage
Not long ago, we built a custom document storage system for a client. One of its more interesting features is that it integrates with an internet fax service so users can select documents in the system and then fax them. For each fax, the software automatically generates a cover page.
To make the cover page, I first made a nice template using a drawing tool and then saved it in PDF format. I then used Zend_Pdf to write over the template with dynamic content. I first demonstrated this technique in this article.
Since the cover page is only used once (during transmission) and easy to regenerate, I don’t save the output to the filesystem. Instead, I create a
php://tempstream. The instance methodZend_Pdf->render()writes the PDF output only to that stream, which is then rewound. Functions likestream_copy_to_streamorfpassthrucan then be used to send the final output where it needs to go, and the whole process normally never needs to use the disk. -
Commodore LCD Firmware October 4th, 2008
The files below are EPROM images from Bil Herd’s prototype Commodore LCD.
You can also download them as a single archive: commodore-lcd-roms.zip
I’ve always been curious about this machine so I sent my device programmer to Bil so that he could read the EPROMs. Bil was very kind to do this and we should all thank him for it. This is the first time that these images have been seen in many years.
Start your disassemblers!
-
PHP Developer Best Practices September 16th, 2008
Matthew Weier O’Phinney and I gave a tutorial session at ZendCon 2008 this year titled “PHP Developer Best Practices”. The tutorial touched on source control, coding standards, testing, documentation, and more.
The slides are now available in PDF format.
We were located in Hall B of the Santa Clara Convention Center, which is a very large room that’s also used for the keynotes. Andi told us the room was selected based on the number of people who registered for our session. We initially had doubts but the attendance was greater than any previous year and the room worked out quite well. We were thankful that unlike last year, everyone was able to get a seat.
Thank you to all who attended. We enjoyed meeting many of you during the breaks and hope that you found our session helpful.
-
Commodore SuperPET September 1st, 2008
I’ve been spending more of my free time recently restoring vintage computer hardware. I am interested in Commodore 8-bit equipment, from the PET/CBM line through to the 64/128 home computers. I think it’s important to preserve computer history to remember the machines that got us where we are today.Since much of the hardware I restore is over twenty-five years old, at least half of it is not working when I receive it. I try to repair everything I can when it’s practical. My Flickr photos page has daily progress of my chip-level repairs on this equipment.
Recently, I received a Commodore SuperPET computer. This is a remarkable machine that was a collaboration between Commodore and the Computer Systems Group at the University of Waterloo in Ontario, Canada. The SuperPET is a standard Commodore PET 8032 computer with an internal expansion that adds a powerful Motorola 6809 microprocessor, an additional 64K of expansion RAM, a fast 6551-based RS232 serial port, and custom Waterloo software in ROM.
The SuperPET can operate in MOS 6502 mode, where it is a Commodore PET 8032 with the extra 64K expansion and 6551 ACIA. Curiously, this 64K expansion memory is not compatible with the 8096. A switch on the side puts the SuperPET into 6809 mode, where it can run a number of disk-based Waterloo programming languages including BASIC, Pascal, APL, Fortan, and COBOL. When in 6809 mode, a menu in ROM prompts the user to select a language which is then loaded from disk.
My SuperPET seemed to work when I got it, with the 6502 mode working perfectly and 6809 mode showing the power-on menu. However, after obtaining the disk-based software, none of the Waterloo languages would run after loading. After verifying the disks were good, I suspected the 64K expansion RAM since the rest of the machine seemed to be working. Using technical information from the PET Index on 6502.org, I wrote several memory test programs to exercise the expansion RAM.
The expansion RAM is comprised of thirty-two 4116 DRAM chips. A couple of these had become loose from their sockets and my test program found that one of them had failed. I got a replacement from the pick-up counter at Jameco and installed it. Now, my SuperPET passes my expansion memory test and also boots all of the Waterloo languages. It is now fully functional and I’m exploring the Waterloo software.
One of the most interesting features of the Waterloo languages is how files are accessed. Commodore disk drives attached to the SuperPET are accessed with a filename like
disk8/1.program-namewhich selects unit 8, drive 1. The SuperPET was designed to be attached to a mainframe computer, known as the “host”, through its serial port. Accessing a file likehost/program-namewould load it from the mainframe if it was running the specialHOSTCMprogram from Waterloo.The
HOSTCMprogram was available for VM/CMS and other mainframe operating systems. Beyond that, I’ve not been able to find out much information about it. I would like to figure out the protocol and write a program so that a modern PC could be used as a host computer for the SuperPET.If you have any information on the SuperPET or have Commodore hardware you’d like to donate, please contact me.
-
Speaking at PyWorks 2008 August 29th, 2008
PyWorks is a new conference from the folks at Python Magazine. It is being co-hosted with a PHP conference, php|works, and attendees of either conference have access to talks on both the Python and PHP tracks. Being a user of both languages, I think this a great idea and I’m looking forward to this format. I’ll be giving two talks at PyWorks 2008:
Microcontroller Simulation in Python
This talk will present Py65, my open-source simulation of a small microcomputer system based on the MOS 6502. The 6502 is a very famous microprocessor that was used in early microcomputers like the Commodore 64, but its design has stood up for over 30 years. Cores based on the 6502 are now widely used in embedded devices, with WDC estimating annual volumes in the hundreds of millions of units.
Using Python and software tools such as Py65, low-level software for embedded systems can be developed and tested much faster. This talk will discuss the design and implementation of Py65, how it and tools like it can help, and will also touch on other Python libraries that can assist with embedded development. (PyWorks Abstract)
URL Mapping with Routes
Routes is a Python package that provides a solution to the problem, “how do I map URLs to my code?” Its solution is an interesting one, and is actually a re-implementation of a feature from Ruby on Rails. Routes itself has also now been ported to PHP 5 as part of the Horde project. Routes is used by the Pylons web framework and other frameworks in Python, and is relatively easy to use as a standalone package.
The Routes method of URL dispatch is based around pattern matching rather than object publishing. For those new to Routes, we’ll have an introduction to the basic Routes concepts and how it works. We’ll also dive into the Routes internals and follow some URLs through their recognition phase, learning about how Routes does it job along the way. Web developers and framework implementers alike will gain a better understanding of Routes and how to use it effectively. (PyWorks Abstract)
-
Speaking at ZendCon 2008 August 22nd, 2008
I will be speaking again this year at the Zend/PHP Conference. I have teamed up with Matthew Weier-O’Phinney to present PHP Developer Best Practices:During this tutorial session, we will cover a number of best practices you can institute in your organization or in your personal coding toolbox, including:
- Testing Strategies
- Coding Standards
- Documentation
- Version Control
- Deployment Methodologies
We will discuss the hows and whys of each topic, and strive to cover concrete examples that you can start implementing immediately, so that you and your developers can start working more efficiently.
It has become a tradition for Matthew and I to present a tutorial session together at ZendCon. We’ve done a new one every year since the first ZendCon. I always enjoy this conference and I’m looking forward to it again this year.
Update: Slides from the presentation are now available.
-
OSCON 2008 Slides July 25th, 2008
Slides from my two talks at OSCON 2008 are now available:
Both talks were well attended had great audience participation. Thanks to everyone that attended and I hope you enjoyed them.
About seven people came up after the Integration Testing talk with good questions and feedback. I think that 45 minutes was enough to provide good starting points for the topics covered but it was clear that these users were engaged and ready to dig into it more. If I give this talk again, it is going to be in a tutorial format so we can get much deeper into the material and have some exercises.
After my Supervisor talk at the end of the day, I had the pleasure of going out to dinner with Roger Hoover and some other Supervisor users. It’s been very exciting to see Supervisor picking up traction over the past year. Besides the work that Chris McDonough and I are doing, Supervisor is being used in several large architectures of companies whose names you know. If you’re using Supervisor and wouldn’t mind us telling others about it, please contact me.
-
Py65: 6502 Microprocessor Simulator July 1st, 2008
I’ve started a new Python project called Py65. It aims to provide building blocks for modeling 65xx microcomputer systems in software, with a focus on homebuilt hardware. Using simulation, embedded systems software can be developed and tested much faster.
In its current form, Py65 supports the original NMOS 6502 microprocessor from MOS Technology. The venerable 6502 and variants of it powered such famous microcomputers as the Commodore 64, game systems like the Atari 2600 and Nintendo Entertainment System (NES), as well as thousands of consumer and embedded devices today.
I haven’t released a package for Py65 yet but I’ll be doing this after some organizational changes are complete. For now, you can get it from its Subversion repository linked from the Py65 page on Ohloh.
The Python interactive interpreter itself can be used as a monitor:
>>> from py65.mpu6502 import MPU >>> mpu = MPU() >>> mpu <6502: A=00, X=00, Y=00, Flags=20, SP=ff, PC=0000> >>> mpu.memory[0x0000:0x0001] = (0x69, 0x16) #=> $0000 ADC #$16 >>> mpu.step() <6502: A=16, X=00, Y=00, Flags=20, SP=ff, PC=0002>
The simulator currently supports all legal NMOS 6502 opcodes and I have written a large unit test suite to verify its core. Most of its tests look like this one:
def test_bne_zero_set_does_not_branch(self): mpu = MPU() mpu.flags |= cpu.ZERO mpu.memory[0x0000:0x0001] = (0xD0, 0x06) #=> BNE +6 mpu.step() self.assertEquals(0x0002, mpu.pc)
Py65’s own unit tests hint at what tests for your own assembly language programs could look like. I am especially interested in unit testing my own embedded software and I will be working on an assertion vocabulary and test helpers for this.
There are a lot of possibilities for what could be done with Py65. For now, I plan to include a system for mapping virtual devices into the microprocessor’s address space, add more hooks into the simulator, and include support for additional 65xx family hardware.
Py65 does not include an assembler. If you need one, I recommend André Fachat’s excellent xa assembler which is currently maintained by Cameron Kaiser.
