edev

One of the difficulties [for Palm developers] is that there's no one killer program Palm owners want to go into a store to buy. In fact, aside from the built-in programs for the Palm, no application is on more than 10 percent of handhelds.

-- CNet article, "Can developers pull dollars from Palms?", Feb. 8, 2002

The Nokia Communicator. The Handspring Treo. The Samsung i300. Combined with about a thousand other mobile phones offering "Wireless Web" access in various shapes and forms, it's clear that the Internet is finally going mobile. The bandwidth is a bit low yet, but that should be changing as it becomes more popular. But the problem is, it's not all that popular. People everywhere are using mobile phones for SMS -- the small-screen equivalent of e-mail -- but the Internet and all its content is getting ignored.

Not for lack of services, mind you. Yahoo! is entirely handheld-friendly, and people still aren't using it in droves, despite the availability of things like weather, stocks, sports scores, movie listings, the latest news and so forth. Part of this is because the screens are so darned tiny, but I don't think that would stop people if they wanted to read badly enough. No, the real problem is that Yahoo! doesn't offer any compelling content online that a mobile phone user couldn't get just as easily by picking up a local newspaper -- either today's or tomorrow's. Sure, I can get sports scores for cities halfway around the globe, but why would people want to? I have Yahoo! on AvantGo on my handheld, but the only thing I ever use it for is to look up movie listings occasionally -- and that's because I'm just too lazy to find the newspaper.

Despite all the hype about the wonders of the wireless Web, the available mobile devices are not able to provide a wireless Web adventure that would impress anybody. Transmission speeds lag behind the dial-up access rates the average AOL user gets at home. And the big question remains, who needs it?

-- NewsFactor.com article, "Where Is the Wireless Web?," Feb. 13, 2002

So portal-type content isn't really driving people to use the Web wirelessly. What would? The kind of content you can't get in a newspaper, obviously -- niche reviews, obscure factoids, interactive discussions. These things are available on the Web, but most of those sites aren't designed for small screens. Browsers for those devices, like Handspring's Blazer browser, try to optimize 800x600-resolution Web pages for 160x160 screens, but they can only do so much. Try surfing the Web using the text-only Lynx browser and you'll see why: complex layouts using tables and JavaScript-dependent features cripple browsers that provide anything less.

The Wireless Web should be the mobile device's "killer app," but it's not. All the truly compelling sites are slow and badly formatted, and all the fast and well-formatted sites lack compelling content. Most sites try to offer as much information and navigation as possible in one page, filling up the left or right columns (or both) of every page with the same stuff and leaving the main page available for specific content. Everything2 does this, too. E2 is generally easy to use with Lynx, but all those wonderful nodelets still sit at the bottom of the page while the Search stays stubbornly at the top, forcing users of non-graphical browsers to scroll and squint to identify either one.

In many of the more relaxed civilizations on the Outer Eastern Rim of the Galaxy, the Hitchhiker's Guide has already supplanted the great Encyclopedia Galactica as the standard repository of all knowledge and wisdom, for though it has many omissions and contains much that is apocryphal, or at least wildly inaccurate, it scores over the older, more pedestrian work in two important respects. First, it is slightly cheaper; and second, it has the words DON'T PANIC inscribed in large friendly letters on its cover.

-- "The Hitchhiker's Guide to the Galaxy" by Douglas Adams

E2 is never going to be popular with mobile users in its current state. But it should be. After all, E2 offers all the things the portal sites don't, and all in one place. Movie and book reviews, by people who aren't paid to hype them. Biographical information on celebrities and historical figures. Popular culture. Definitions. Textbook knowledge. Famous poetry and prose, both original and not. Humor. Blogging. Eclectica.

H2G2 tries to be what Douglas Adams' Hitchhiker's Guide wanted to be, but fails for a number of very good reasons. Plus, E2 offers a community spirit and a very open system for adding and editing content. At the time of this writing, E2 has nearly half a million distinct writeups by some 40,000 distinct users. And none of it is available in any other, single location.

We are the killer app. The world just doesn't know it yet.

I am pleased and proud to announce E2's new better XML output. This increases our current set of XML ticker functionality, plus provides a better and cleaner set of XML (well formed, but not yet validating) displays for this site.

clientdev: new XML ticker output, by JayBonci

I would like to put forth a vision for all the developers who use and enjoy E2: Build a wireless client to the full database that is small, fast, and easy to use. And, of course, freely available to anyone who wants to use it. This would be the killer app for wireless handhelds throughout the English-speaking world.

Imagine, for a second, being able to look up honest movie, music or book reviews right in the store before you buy. Or accessing a travel guide to New York City while you're waiting for the subway. Or getting some fun information on your favorite sports team's opponent rather than just statistics from last season. Or hearing a strange piece of slang while visiting a distant city, or a technical term while reading a magazine article, and being able to look it right up. Or just wanting something random to read until your class starts or your meeting ends without having to download it first. Imagine how much the rest of the world would like to be able to do all that, too.

But E2 isn't good for wireless use yet, because it requires a Web browser and because it's too friggin' slow. I don't mean server loads. I mean the nodes that contain too many writeups with too much text for a slow mobile modem to handle, the search tools that are too far away for a small screen, the soft links and nodelets that provide quick navigation for a user of a graphical browser, but just slow things down needlessly for those who don't.

E2 has all the content it needs to be a wireless hit. It just needs an optimized interface.

---

Things To Do

These are the major issues I think a wireless, mobile version of E2 would need. Many of them are things that would or should never be done on the graphical, PC-based version of E2. Nevertheless, a different environment requires certain changes.

• Forget the Wireless Web. A slimmed-down version of E2 for portable browsers is not the answer. Certain elements, like the Search bar, need to be on-screen at all times or at most a single click away. A user who has his User Preferences set up one way for the graphical E2 might want them a completely different way for the mobile E2. What we need is an entirely new client that accesses E2 using XML, and then displays it in a interface designed for small screens.

• Trim the Extras. Nodelets must go. Softlinks must go. Useful features like adding a writeup or gabbing in the Chatterbox should be available by selecting a menu or clicking a fixed, non-scrolling button. Bookmarks of favorite writeups should be tucked away in their own menu. The text display of the client should display nothing but writeup headers and text, using the same HTML formatting that we support now.

• Optimize the Writeup Display. E2's current node layout is great for the WWW, bad for the Wireless Web. Getting several writeups at once would be prohibitively time-consuming on low-bandwidth, mobile connections. Only one writeup should be displayed at a time, with single-button access to the last or next writeup. Writeup titles should be static while writeup headers and content should scroll. Voting up or down should be as easy as pressing or tapping two buttons (one would be prone to accidents). Writeup headers should be kept to the bare minimum -- author, type, date, and reputation.

• Get It Everywhere. PalmOS. PocketPC. Whatever the Nokia Communicator uses. As long as they're looking for developers and "killer apps" to sell their new wireless devices, we should take them up on it. This client should work on every popular mobile wireless device that's on the market, and it should work as identically on every one of them as possible. Obviously, different screen sizes and resolutions (especially the elongated Communicator screen) will require some major changes to the interface layout, but there's no reason we can't keep the same on-screen tools and menus, just stored in different places. UPDATE: There's a book on the way titled Flash Enabled: Flash Design and Development for Devices (http://www.amazon.com/exec/obidos/ASIN/0735711771) that discusses designing Macromedia Flash for portable devices. I'm convinced that Flash, especially the newly-released Flash MX, will be the way to go. Many smart phones can use it, PocketPC can use it, and I believe it's coming to the PalmOS very soon. With built-in XML support, this will be the fastest way to produce a multi-platform client (if not exactly the most backwards-compatible).

• Let It Make Money. Yes, by all means, give the client away, especially to device manufacturers. And keep E2 on the WWW free for everyone. But the mobile client is a special "extra," and if it becomes popular, it should offer the opportunity to make money for its developers and servers. If we're going to tax the server extra with this, we should get something back, right? So let the client be a 30-day trial, or think of how to sell $2-a-month subscriptions to access the content with it. Or else develop a whole new way to sell ad space on the thing -- let Barnes and Noble put a blurb beside every writeup containing the word "novel," for instance. I can call anyone cheaply from my home phone, but I pay a privilege to do it on the go; it's fair and reasonable for E2 to expect the same.

• Involve Your Friends. Yeah, maybe you're whiz-bang enough to develop something like this on your own, but why when you can get help? Join the clientdev group here on E2, find out what you need, stick it up on SourceForge, and do your darnedest to let everyone who owns a mobile device know what it is and what it does. They don't have to know or like E2. They don't even have to see the word "Everything2" anywhere on it. They just have to use it and love it. And if they offer suggestions, write 'em down and try to add 'em in. Again, just because E2 on the WWW doesn't do something a certain way doesn't mean a wireless client can't.

Who We'll Need

• Anyone familiar with the development tools for existing mobile devices, especially PalmOS, and/or anyone willing to learn how they work.
• Anyone familiar with XML and how to build and parse it, and/or anyone willing to learn.
• Edev members to build a few new doctypes that the mobile client can access. Existing doctypes are designed for the graphical WWW; we need new ones just for mobile clients. (Example: a new type of node document that contains only one writeup and a link to previous/next ones.)
• E2 gods who can make the above things happen on the server side.

Interested? Join clientdev and tell us what you can do.

m_turner's idea is a good start, but the magic-number-handling can all be done server-side, instead of client-side. Enter hash function. Basically, a hash function is a function in the form f(x), but the length of f(x) remains constant, no matter what x is. What that means is, you pass a value to the hash function. It returns another value, the hash digest, whose value depends on what you pass it. But, the length of what it returns is always the same, and is usually less than the length of what you pass it. So, it is possible to have two messages with the same hash digest, but it is very hard to make a message that matches to a certain hash digest.

So, what? Well, what happens is the hash digest of each message is stored with the message in the database (or in a session cookie?). If two messages have the same hash digest and are sent within an hour of each other it is almost definite that they are identical messages. If a message is deemed to be identical, it is rejected. To summarize:

1. Bob sends a message
2. The hash function gives us the hash digest of Bob's message
3. The hash digest of the message is stored with the message
4. Bob sends another message with the same hash digest, within an hour of the previous message
5. Bob's second message is rejected

1. Joe sends a message
2. The hash function gives us the hash digest of Bob's message
3. The hash digest of the message is stored with the message
4. Joe hits the Back button and sends another message
5. The hash function gives us the hash digest of Bob's message
6. The hash digest of the message is stored with the message
7. The two hash digests do not match
8. Joe's second message is allowed

Or, like I said, we could use session cookies:

1. Bob sends a message
2. The hash function gives us the hash digest of Bob's message
3. The hash digest of the message is stored in a cookie which will expire at the end of the session
4. Bob sends another message before the session expires
5. The hash digest of the second message is the same as the one in the session cookie
6. Bob's message is rejected

Personally, I like session cookies, since they won't take up space in the database.

---

m_turner points out that not all clients that can send cookies support cookies (for example, the Java Chatterbox). However, most clients that cannot receive cookies (like the Java catbox) cannot refresh or reload or anything like that. m_turner also points out that a session cookie doesn't necessarily have a time limit. In that case, we can set the cookie to expire in, oh, say, half-an-hour from when it is saved. m_turner's final idea is to put a constraint in the database itself that doesn't allow the recipient and message to be the same for two or more messages. This works fine, but we should see which is faster: computing and then checking hash digests, or just checking the message body itself.

I am pleased and proud to announce E2's new better XML output. This increases our current set of XML ticker functionality, plus provides a better and cleaner set of XML (well formed, but not yet validating) displays for this site. This will eventually replace our current displaytype=xml output, but for a while the old tickers will be in place, and the old displaytype=xml interface will still be good (for how long remains to be seen of course).

In general these tickers and interfaces are to be treated as if they were in beta, as they still are. Interfaces are subject to change, so please consider any code of this nature "in beta". Interface changes will be mentioned here, but as time goes on, these interfaces will firm up. Please make your code flexible, as one of the large items that is subject to change is the interpositioning of elements within the XML tags, and new elements will be added (especially to the user element).

One requirement of all clients is to assume a server error on malformed elements. This is because of in case of an actual server error (yes, we all know how perfect our code runs and how infrequently one of the boxes does not run out of memory), the output will in fact be mangled. If you encounter a server error, please try to reload the page in a minor amount of time, and fire a decent user error off. This is a general limitation to ecore in the sense that this is only xml-ish behavior, and does not reflect any underlying changes to the core.

It should not matter which user the client uses. Preferably, we'd only like to have logged in users be able to really access the ticker and get at important information, but in reality, Guest User should have no problem accessing each node.

Document text (in large blocks), should always be encodeHTML'ed. Otherwise it will form badly formed output (see the link parsing section below). This is bad. Yes, you may have to go through the trouble of taking ampersands back to their normal form, and other escaped characters, but that should be doable without much trouble. The goal of this export was to create readable output that any parser on the street could read, so all the characters need to be escaped, no matter what.

The new basic interface is displaytype=xmltrue. This produces valid output for the following nodetypes:

• user
• writeup
• e2node

Each of these provides a different set of functionality, but in general quite closely mimics the actual interface of Everything2. Many types of documents can not, and will not be supported. If you try to xmltrue a document that is not supported, you should be getting, at the very least, a "no valid conversion" document (info tags). This means that simply, you can't view it. You should always, whenever available, view nodes by node_id. This will give you a cleaner viewing style across E2. If you need to look up a node title, use the XML Search Interface. It's fast, and it's clean. It should work well for you (hopefully).

All nodes have a standard header. Writeup is an embedded type, even though it duplicates some information (rather than having to special case it in to a bunch of things, think of a writeup as an e2node with only one writeup in it. In a lot of ways, that's how it works on e2, and that's how the data stream works for code cleanliness. Each node should give you it's node_id, title, author name, author_id, nodetype, creatime, etc. The XML export is designed with the thought that you should not have to keep a large client side mapping of node_ids to names. That would be largely inefficient, so we provide you with that information whenever possible. If forced to make a choice between the two, we should always give you the node_ids (except in the case of something like Personal Nodelet where the strings are user editable, and have no bearing on what may or may not actually exist (anymore at least), in the database.

• Standard header information is provided by xmlheader
• Standard footer information is provided by xmlfooter (at this point, the xmlfooter output is really lame, but that's unimportant.
• The generic xml request is handled in ecore by node xmltrue page
• formxml handles the request type negotiation. This was chosen over an indivual displaypage for each type, as it would simplify the process for now. We might move over to such a system later, but it allows us a little more ease in making changes for the time being.

displaytype=xmltrue (user)
Currently, homenode exporting works fairly well. You get the userstrings, the document text (the homenode text), parsed correctly for the usertype, and the experience. Also you can view their bookmakrs if they have them available. The bookmarks use a standard link mechanism known as <e2link> This allows them to have a standard interface for things such as firmlinks, softlinks, and bookmarks. They contain the node_id and inside, for their #PCDATA, they have the title, again, saving you from the node_id => title mapping that we fear.

...in the future they might have user specific information per session, but that has yet to really be nailed down. The exact implementation details of a lot of that are still largely unavailable.

• formxml_user is largely responsible for the generation of the user page.
• displayUserText was modified to help with the output and removal of links. See link parsing below.

displaytype=xmltrue (e2node)
E2nodes export their firmlinks (as <e2link>'s), each writeup as an embedded object and then their softlinks. They also pose a suggestion, if there is also a user under the exact same title, the same as what they normally do now. The number of softlinks are correct and exact for the usertype. This is handled by the softlink htmlcode itself, and thus produces the same output (except not in tabular format). For more information on the writeup portion of it, see writeup below.

• Softlinks are governed by an argument to that htmlcode, and return output if that argument is valid
• xmlfirmlinks does the firmlinks and is largely very similar to:
• xmlwriteup is the generic writeup display code

displaytype=xmltrue (writeup)
The writeup xmltrue display is largely similar to the e2node display, except that you only get one writeup for sure, and a lot of the information is duplicated. One thing that is added to the actual <writeup> tag is the voting information that you get if you either own the node, or have voted on it before. C! Information is not available yet, but has been bugged, and will be eventually added to this type. You get the writeup type information (as it could be non-standard, presumably). Softlinks and firmlinks should be exported per normal on a writeup.

• xmlwriteup, as mentioned earlier, governs the generic writeup output
• formxml_writeup does the combining, and largely only calls xmlwriteup for a good portion of it

Stopping of link parsing
In all three acceptable nodetypes, you can halt the processing of links by appending the following to the url: links_noparse=1; If you don't pass it one, then an "undocumented result happens". Just pass it one. Thanks. This is special cased in to each section and is not specific to any one thing. It will also affect your homenode if you do it there (not in an xml viewtype). This may even work on older exports, but this does not work on any of the tickers. Tickers are being generally overhauled as mentioned below.

The New Ticker type, and it's benefits
The old tickers were derived from nodes with the fullpage type. This still means that their links get parsed. This is largely bad because URLs don't really make any sense to XML clients and was producing, among other things, badly formed content. Therefore, we have created a new type ticker that does everything the previous one does, except parse links from the text field areas (we do encodeHTML them, but that's to be expected). Since hyperlinks depend per implementation, we don't need to force clients into writing a scrubber for the URLS. Also, we want to create the tickers, so that we are not based performance wise on their implementations of the tickers (IE, cached tickers), but please feel free to make your clients as kind as possible. There are currently three "tickers" available:

• Random Nodes XML Ticker (by request)
• Scratch Pad XML Ticker (also by request)
• E2 XML Search Interface (large props to nate's RDF Search, which was easy enough to port and extend).

Random Nodes XML Ticker
The random nodes ticker works really similarly to the random nodes nodelet. You get 12 random nodes (and even a bit of wit), ever minute or so. It gets fed off a nodelet called Random Nodes XML Feed, so feel free to hit this nodelet as often as you like, it's quite cheap, I can assure you. It's also fine to hit from Guest User.

Scratch Pad XML Ticker
The scratch pad ticker merely dumps out the contents of your scratch pad, and whether it is public or private. It should return an empty pad if the user is guest user. It does not take any input for change (but might someday).

E2 XML Search Interface
This should be where your client spends a lot of it's time. It takes two parameters "keywords" and "typerestrict". These parameters take a keyword string (separated by spaces, exactly how you'd type it in to the search form, and a typerestrict parameter, which you'd pass it a string of which type to restrict against (user or e2node are really the only two sensible choices). If you do not pass a typerestrict parameter, the script will default to an e2node. It will return a large page or results, similar to the Findings: page on E2. Please do not search in the standard way for that node, but use this ticker to gather your intelligence about what's available for viewing in any client. It returns a list of <e2link>'s with titles and node_ids. If you pass it a typerestrict you can also do inexact searches on users. You will get only the type that you passed it in (this will be available inside the XML you get back.

---

This is still a work in progress, as largely most of these interfaces are finished or in beta quality. If you see any typos or things that could use to be clarified, please let me know. There are bugs listed offsite on the sourceforge page for the e2client project insanefuzzie is working on (and I am merely helping to port). Please check there before reporting a bug, but also feel free to suggest anything that might make usage of these interfaces easier. We plan on doing an all out schema for these interfaces, but vailidity is by far a low priority, as we are using all internal apps to parse them. Hopefully this will prove useful to client developers, and in turn, the general populace of E2.

**Note, I typed this all up, very very tired, so if the sentences don't make any sense, please let me know. This is a prelimiary document for this new feature.

How to go from Linux to the finest Content Management System ever:
step by step.

   Everything2.com runs on an engine called eCore, created by our friends over at the Everything Development Company. It can be a bit tricky to setup your first time through, but don't give up! This is, in my opinion, one of the best content management engines around, and with a basic understanding of perl you can begin programming on it in minutes.

   I happened to have a spare Macintosh at work with a clean install of Debian on it, so all the examples from this node will be using Debian 2.2 on a powerpc kernel. So far as I've seen, this method also works on the testing and unstable trees of debian, on any platform with Apache, MySQL, and perl compiled for it.

Materials needed:

• One working computer with a fully-installed copy of Linux on it (these examples will be based on Debian).
• One connection to the internet (or a copy of the needed debian packages and perl modules on disk or CD)
• Root access on the computer, via su or sudo.
• About ten minutes, if all goes as planned.

Procedure:
Your keystrokes are in bold monospace,
The computer's responses are in regular monospace.

Step one is to get all our tools lined up. We will need working copies of Perl, MySQL, Apache, mod_perl, DBI, CGI, and a handful of other perl modules. Many of the functions that require root access can be taken care of with su, but I prefer to work with sudo. Sudo is a good utility to have around if your eCore based site ever requires multiple administrators. Since sudo doesn't come pre-installed on debian's stable distribution, we need first to set it up:

danne@local:~$ su
Password: (type your root password here)
root@local:~/home/danne/# apt-get install sudo
Reading Package Lists... Done
Building Dependency Tree... Done
The following NEW packages will be installed:
  sudo
0 packages upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 128kB of archives. After unpacking 340kB will be used.
Get:1 http://http.us.debian.org stable/main sudo 1.6.2p2-2 [128kB]
Fetched 128kB in 1s (80.5kB/s)
Selecting previously deselected package sudo.
(Reading database ... 20897 files and directories currently installed.)
Unpacking sudo (from .../sudo_1.6.2p2-2_powerpc.deb) ...
Setting up sudo (1.6.2p2-2) ...
No /etc/sudoers found... creating one for you.
root@local:/home/danne# visudo

   When in visudo, goto line that says "root ALL=(ALL) ALL", type 'A', press return, type in the username you're currently logged in as, type " ALL=(ALL) ALL", press escape, type ":wq", and press enter. Now, you should be out of visudo, and have it configured to grant the current user root access. Type 'exit' to return to normal user mode.

   Now that sudo's installed, we can finish setting up the base system (things like Apache, MySQL, and some perl modules). To grab some of these files, we'll need a program called wget. Wget isn't installed by default, so we need to install it. Debian makes this easy:

danne@local:~$ sudo apt-get install wget

We trust you have received the usual lecture from the local System
Administrator. It usually boils down to these two things:

        #1) Respect the privacy of others.
        #2) Think before you type.

Password:(the current user's password.  NOT the root password)

   Wget should now be installed, we can go ahead and pull eCore's install files from www.everydevel.com, the necessary perl modules from www.cpan.org, and the neccessary debian packages from http.us.debian.org. These URIs are fully operational af of the time of this writing, but there is a possibility that the ones for www.cpan.org may change. If so (if, after typing 'ls', TermReadKey or File-Spec don't show up) go to http://search.cpan.org and grab the latest URI for the module.¹

danne@local:~$ mkdir ecore
danne@local:~$ cd ecore
danne@local:~/ecore$ wget http://www.everydevel.com/downloads/current-everything.tar.gz -q
danne@local:~/ecore$ wget http://www.everydevel.com/downloads/libmail-sender-perl_0.7-1.deb -q
danne@local:~/ecore$ wget http://www.cpan.org/authors/id/J/JS/JSTOWE/TermReadKey-2.17.tar -q
danne@local:~/ecore$ wget http://www.cpan.org/authors/id/R/RB/RBS/File-Spec-0.82.tar.gz -q
danne@local:~/ecore$ wget http://www.everydevel.com/downloads/apachediffs/access.conf.diff -q
danne@local:~/ecore$ wget http://www.everydevel.com/downloads/apachediffs/srm.conf.diff -q
danne@local:~/ecore$ ls
File-Spec-0.82.tar.gz    access.conf.diff           libmail-sender-perl_0.7-1.deb
TermReadKey-2.17.tar.gz  current-everything.tar.gz  srm.conf.diff

danne@local:~/ecore$ sudo apt-get install mysql-server apache-perl \
      libdbd-mysql-perl libxml-perl libxml-generator-perl libxml-parser-perl \
      libdbi-perl libcgi-perl libxml-dom-perl libapache-dbi-perl mailtools
Reading Package Lists... Done
Building Dependency Tree... Done
Sorry, apache-perl is already the newest version
Sorry, libdbi-perl is already the newest version
The following extra packages will be installed:
  expat libxmltok1
The following NEW packages will be installed:
  expat libcgi-perl libdbd-mysql-perl libxml-dom-perl libxml-generator-perl libxml-parser-perl
  libxml-perl libxmltok1 mysql-server libapache-dbi-perl
0 packages upgraded, 9 newly installed, 0 to remove and 0 not upgraded.
Need to get 1437kB of archives. After unpacking 4068kB will be used.
Do you want to continue? [Y/n] Y
Get:1 http://http.us.debian.org stable/main expat 1.1-1 [9350B]
Get:2 http://http.us.debian.org stable/main libcgi-perl 2.76-17 [125kB]
Get:3 http://http.us.debian.org stable/main libdbd-mysql-perl 1.2202-4 [106kB]
Get:4 http://http.us.debian.org stable/main libxml-dom-perl 1.25-1 [110kB]
Get:5 http://http.us.debian.org stable/main libxml-generator-perl 0.5-1 [8836B]
Get:6 http://http.us.debian.org stable/main libxml-parser-perl 2.27-3 [297kB]
Get:7 http://http.us.debian.org stable/main libxml-perl 0.05-1 [100kB]
Get:8 http://http.us.debian.org stable/main libxmltok1 1.1-1 [47.1kB]
Get:9 http://http.us.debian.org stable/main mysql-server 3.22.32-6 [633kB]
Get:10 http://http.us.debian.org stable/main libapache-dbi-perl 0.87-1 [38.6kB]
Fetched 1475kB in 10s (141kB/s)
Configuring packages ...

   At this point, depending on how debconf is setup for you, you should get some configuration dialogs. Many of them will be notices (requiring you only to press ok), but depending on what has to install, you may see more than that. Most of the questions should be very straightforward.

Selecting previously deselected package libxmltok1.
(Reading database ... 20879 files and directories currently installed.)
Unpacking libxmltok1 (from .../libxmltok1_1.1-1_powerpc.deb) ...
Selecting previously deselected package expat.
Unpacking expat (from .../expat_1.1-1_powerpc.deb) ...
Selecting previously deselected package libcgi-perl.
Unpacking libcgi-perl (from .../libcgi-perl_2.76-17_all.deb) ...
Selecting previously deselected package libdbd-mysql-perl.
Unpacking libdbd-mysql-perl (from .../libdbd-mysql-perl_1.2202-4_powerpc.deb) ...
Selecting previously deselected package libxml-parser-perl.
Unpacking libxml-parser-perl (from .../libxml-parser-perl_2.27-3_powerpc.deb) ...
Selecting previously deselected package libxml-dom-perl.
Unpacking libxml-dom-perl (from .../libxml-dom-perl_1.25-1_all.deb) ...
Selecting previously deselected package libxml-generator-perl.
Unpacking libxml-generator-perl (from .../libxml-generator-perl_0.5-1_all.deb) ...
Selecting previously deselected package libxml-perl.
Unpacking libxml-perl (from .../libxml-perl_0.05-1_all.deb) ...
Selecting previously deselected package mysql-server.
Unpacking mysql-server (from .../mysql-server_3.22.32-6_powerpc.deb) ...
Selecting previously deselected package libapache-dbi-perl.
Unpacking libapache-dbi-perl (from .../libapache-dbi-perl_0.87-1_all.deb) ...
Setting up libxmltok1 (1.1-1) ...
Setting up expat (1.1-1) ...
Setting up libcgi-perl (2.76-17) ...
Setting up libdbd-mysql-perl (1.2202-4) ...
Setting up libxml-parser-perl (2.27-3) ...
Setting up libxml-dom-perl (1.25-1) ...
Setting up libxml-generator-perl (0.5-1) ...
Setting up libxml-perl (0.05-1) ...
Settung up libapache-dbi-perl (0.87-1) ...
Setting up mysql-server (3.22.32-6) ...
Starting MySQL database server: mysqld.

Step 2 is to load all that mess we just downloaded and set it all up. We'll be configuring Apache and our perl modules, and getting eCore ready to go. Patching the Apache config files requires root access, but I've had trouble getting sudo to do this. In this example, we'll just use su:

danne@local:~/ecore$ su
Password: (enter the root password here)
root@local:/home/danne/ecore# cat srm.conf.diff | patch /etc/apache/srm.conf
patching file `/etc/apache/srm.conf'
Hunk #1 FAILED at 24.
1 out of 2 hunks FAILED -- saving rejects to /etc/apache/srm.conf.rej
root@local:/home/danne/ecore# cat access.conf.diff | patch /etc/apache/access.conf
patching file `/etc/apache/access.conf'
root@local:/home/danne/ecore# exit
exit

    Yeah, that 'Hunk #1 FAILED' looks pretty ominous, but it doesn't get in the way of getting things setup and functioning.
    Next, we have to install and configure our three perl modules. The first of which has been packaged as a deb file, making installation a snap. The other two (like all perl modules) aren't that difficult to install, though:

danne@local:~/ecore$ sudo dpkg -i libmail-sender-perl_0.7-1.deb
Password: (enter user's password here, not root.)
Selecting previously deselected package libmail-sender-perl.
(Reading database ... 21162 files and directories currently installed.)
Unpacking libmail-sender-perl (from libmail-sender-perl_0.7-1.deb) ...
Setting up libmail-sender-perl (0.7-1) ...
danne@local:~/ecore$ tar zxf File-Spec-0.82.tar.gz
danne@local:~/ecore$ cd File-Spec-0.82
danne@local:~/ecore/File-Spec-0.82$ perl Makefile.PL
Checking if your kit is complete...
Looks good
Writing Makefile for File::Spec
danne@local:~/ecore/File-Spec-0.82$ make

   Look for errors here, and after every time you run 'make'. There will be alot of text floating by, but I'm not going to list it all here. Unless 'perl Makefile.pl' failed, this shouldn't fail. On the next line, we include 'UNINST=1' because the file::spec module, depending on your version of debian, might or might not be there. For the sake of consistency, we'll install the newer version:

danne@local:~/ecore/File-Spec-0.82$ sudo make install UNINST=1
   (you will see output here, but, as it varies system to system, I've left it out.)
Writing /usr/lib/perl5/5.005/powerpc-linux/auto/File/Spec/.packlist
Appending installation info to /usr/lib/perl5/5.005/powerpc-linux/perllocal.pod
danne@local:~/ecore/File-Spec-0.82$ cd ..
danne@local:~/ecore$ tar zxf TermReadKey-2.17.tar.gz
danne@local:~/ecore$ cd TermReadKey-2.17
danne@local:~/ecore/TermReadKey-2.17$ perl Makefile.PL
Checking if your kit is complete...
Looks good
Writing Makefile for Term::ReadKey
danne@local:~/ecore/TermReadKey-2.17$ make
   (many lines omitted.  Don't forget to watch for errors!)
danne@local:~/ecore/TermReadKey-2.17$ sudo make install
   (lines once more omitted.)
danne@local:~/ecore/TermReadKey-2.17$ cd ..

   Look where it installed your perl modules (*.pm files) to. You'll need that later to patch up eCore's XML.pm.
   FInally, we get to install the eCore modules:

danne@local:~/ecore$ tar zxf current-everything.tar.gz
danne@local:~/ecore$ cd everything/
danne@local:~/ecore/everything$ perl Makefile.PL
Install directory [/usr/local/everything]: (press enter)
May I append:

        Include /usr/local/everything/everything.apache.conf

to your httpd.conf file? (N/y)Y
Where is your Apache's httpd.conf? [/etc/apache/httpd.conf](press enter)

Everything is configured to be installed as follows:
  - Install directory: /usr/local/everything
  - Append Include to: /etc/apache/httpd.conf

Is this correct? [y]:(press enter)
Warning: prerequisite XML::DOM 1.27 not found at (eval 1) line 220.
Checking if your kit is complete...
Looks good
Writing Makefile for Bundle::Everything
Writing Makefile for Everything

   The XML::DOM warning comes on debian's stable distribution only (so far as I've seen), because the installed version of libxml-dom-perl is 1.25-1. Everything seems to work fine with that version installed, however (no pun intended).

danne@local:~/ecore/everything$ make
    (Lots of text excluded.  Just watch for error messages)
danne@local:~/ecore/everything$ sudo make install
    (Lots of text excluded.  It should end with something to this effect:)
Installing /usr/local/lib/site_perl/Everything.pm
Installing /usr/local/lib/site_perl/Bundle/Everything.pm
Installing /usr/local/man/man3/Bundle::Everything.3pm
Writing /usr/local/lib/site_perl/powerpc-linux/auto/Everything/.packlist
Appending installation info to /usr/lib/perl5/5.005/powerpc-linux/perllocal.pod

   Next, we need to patch up XML.pm. Due to a glitch in the XML postprocessor, nbmasta won't import nodeballs into the database. Remember where it installed the perl modules? I've seen it install them in /usr/local/lib/site_perl/Everything and /usr/local/share/perl/5.6.1/Everything. When you earlier typed 'sudo make install', it told you where it was installing. We need to go to this directory:

danne@local:~/ cd /usr/local/lib/site_perl/Everything
danne@local:/usr/local/lib/site_perl/Everything$ sudo vi XML.pm

   We need to find this:

sub unMakeXmlSafe {
        my ($str) = @_;

        $str =~ s/\&lt\;/\</g;
        $str =~ s/\&gt\;/\>/g;
        $str =~ s/\&amp\;/\&/g;
        return $str;
}

   And insert this:

sub unMakeXmlSafe {
        my ($str) = @_;

        $str =~ s/\&lt\;/\</g;
        $str =~ s/\&gt\;/\>/g;
        $str =~ s/\&amp\;/\&/g;
        $str =~ s/\&quot\;/\"/g;
        return $str;
}

Step 3 is to setup an eCore based site and restart Apache and MySQL so they will see their new configurations and users:

danne@local:~$ cd /usr/local/everything/
danne@local:/usr/local/everything$ sudo bin/install_esite (desired site name)
found Everything install at .
found Everything install at /usr/local/everything
Ok to use /usr/local/everything for Everything directory? (N/y)Y
Please enter the root password for mysql (MySQL password.)
Creating new database 'myNewESite'
Which SQL user do you want to control 'myNewESite' [root] esite

Please specify the password for this user: [none] (press enter)
Please specify the host for this user [localhost] (press enter)
User 'esite' created!
Where is your web directory? This directory should be accessable by apache
[/var/www] (press enter)
Would you like me to create an /var/www/incoming directory where you can upload 
images and files? (N/y) Y
What user does Apache run as? [nobody] www-data
/var/www/incoming created and chown to www-data!
Installing nodeball.  Hang on.
Creating tables...
   - Done.
Skipping already installed table node!
Skipping already installed table nodetype!
Skipping already installed table setting!
Skipped tables have the right columns, though!
Installing nodetypes...
Fixing references...
   - Done.
Installing nodes...
Fixing references...
   - Done.
setting up the typeversion table...

Please set the root user password (max 10 chars): (desired password)
Please confirm the root user password: (...and again)
/usr/local/everything/everything.apache.conf has been modified.  
    Remember to restart the Apache server.

Also note, this change will not take effect unless you have:
        Include /usr/local/everything/everything.apache.conf

somewhere in your httpd.conf


Installation complete, now restart apache with:

        apachectl restart

danne@local:~/usr/local/everything$ sudo apachectl configtest
Syntax OK
danne@local:~/usr/local/everything$ sudo apachectl restart
/usr/sbin/apachectl restart: httpd restarted
danne@local:~/usr/local/everything$ sudo /etc/init.d/mysql restart
Stopping MySQL database server: mysqld.
Starting MySQL database server: mysqld.
danne@local:~/usr/local/everything$ exit
exit

   If all went well, you should be able to point a browser at that computer's IP address, domain name, or http://localhost/ (on that machine only) and see "Welcome to Everything!" with the familiar login nodelet, among others. From there, login as root, with the password you set with install_esite, and have fun!

Security Concerns

The way this is setup is stable, but very insecure. Before you get to having too much fun with your new eCore site, you need to do some MySQL maintainance:

danne@local:~$ mysql -u root mysql
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 3 to server version: 3.22.32-log

Type 'help' for help.

mysql>update user set Password=password('(desired password)') where User='root';
Query OK, 2 rows affected (0.01 sec)
Rows matched: 2  Changed: 2  Warnings: 0

mysql> update user set Password=password('(desired password)') where User='esite';
Query OK, 1 row affected (0.00 sec)
Rows matched: 1  Changed: 1  Warnings: 0

mysql> grant all privileges on myNewESite.* to esite@localhost;
Query OK, 0 rows affected (0.01 sec)

mysql> exit
Bye
danne@local:~$ sudo /etc/init.d/mysql restart
Stopping MySQL database server: mysqld.
Starting MySQL database server: mysqld.
danne@local:~$exit
exit

   These aren't the only security concerns, but they're a good start. Once you're in your site, immediately setup a normal user account and use it for regular operations (doing everything as root can be nice, but you run the risk of deleting something important.)
   If you're concerned about securing your eCore installation, everydevel.com would be a good place to direct questions to. jaybonci or myself would probably be willing to help you out, as well.

---

¹ - I realize this might be more easily done using cpan.pm, but Configuration of that alone would take a full node to explain. This method is simpler.

---

N.B. - After setting eCore up more times than I can count, I've found this manner of doing it to be the simplest and most reliable. If anyone know a faster, simpler way to do this, please /msg me and I'll make a note of it here.

Thanks!

The current form for message post is something like:

<FORM METHOD="POST"
  ENCTYPE="application/x-www-form-urlencoded"
  NAME="formcbox">
(stuff)
<a name='chatbox'></a>
<small><input type="hidden" name="op" value="message"><br /></small>
<INPUT TYPE="text" NAME="message"  SIZE=15 MAXLENGTH=255>
<INPUT TYPE="submit" NAME="message_send" VALUE="talk">
</FORM>

(please note - double quotes should be used for <a name='chatbox'></a>

Insert a random value (or sequential) in a hidden form tag. This is not overly important. This number is re-generated each page load. Thus, the form is:

<FORM METHOD="POST"
  ENCTYPE="application/x-www-form-urlencoded"
  NAME="formcbox">
(stuff)
<a name='chatbox'></a>
<small><input type="hidden" name="op" value="message"><br /></small>
<INPUT TYPE="hidden" NAME="magic" value="42">
<INPUT TYPE="text" NAME="message"  SIZE=15 MAXLENGTH=255>
<INPUT TYPE="submit" NAME="message_send" VALUE="talk">
</FORM>

In the message table, the magic number would be inserted too. This value must be unique within the table. If someone reloads the page then the same magic number is submitted. The second time, it would be rejected.

This does pose several problems:

• Other clients would need to have this magic number submission
• Doesn't let people hit the 'back' button and possibly send a different message if there was a typo to be corrected.

---

A better solution is to actually put in the message table a constraint around the person/message combination so that it must be unique.

Assume:

Name         Null?    Type
------------ -------- ----------------------------
USER         NOT NULL number
MESSAGE      NOT NULL varchar(255)
TIMESTAMP    NOT NULL datetime

In this case, a unique index (its a small enough table that gets cleaned regularly) is placed on the user/message rows:

CREATE UNIQUE INDEX no_chatbox_dups ON chatbox_tab
    ( USER, MESSAGE )

Thus, any insertion of the same user/message information will come back with the error "cannot insert duplicate value into table" which would then need to be handled properly by perl (in this case, silently fail).

Note: this takes advantage of the regular cleansing of the chatterbox by other processes that roll messages out of the table.

---

Session cookies and a hash work for almost every case - people use browsers in the vast majority of cases for chatting on edev. However, there are a few distinct times when this would not work.

With the current wave of E2 clients (and the existing ones that are already exist such as the java chatterbox), not ever client that can send a cookie can properly receive a cookie. Thus, sending the "hash of last message" id to the java chatterbox (or a perl command line chatterbox poster, or the #everything chatterbot relay) will not be picked up. The next message would send out the same cookie as it did before without any such hash.

Requiring this part of the cookie would meet with some resistance for much the same reason the "magic" value did - far too much work on the client side to properly pick up and interpret the information.

Furthermore, there is no 'time-limit' on this session cookie. Consider a person trying to say something when the chatterbox is dead silent.

<m_turner> Hello? Anyone out there?

(5 min pass, the chatterbox is clear again)

<m_turner> Hello? Anyone out there?

In my (probably not so humble opinion) the best solution is one that of the unique constraint upon the chatterbox table itself which is then completely done on the database side without any requirements for how the information got there.