edev

Content syndication isn't a new idea altogether. There are a lot of sites (including slashdot et al) that have rdf feeds and other "tickers" that allow you to syndicate content to other places. If you want a slashdot or everything2 "What's new" box on your homepage, it's really quite easy. With a bit of XML::Simple and basic knowledge of CGI/mod_perl you can cough one out.

What s_alanet has put together is really quite interesting. It has a lot of features of E2 and has some interesting ideas as to the right way to syndicate content, but what's the long term effect of this? It is tough to imagine a website that exists completely as a collection of content hubs (to use his language), working as mirrors of the whole. There are a lot of issues with breaking down a website into a distributed application as such.

The first of which is authentication. Now you can always shortcut this and say that this is guest user only, but that severely strips out the feature set that is available. In essence, you are using a third party to authenticate. It's like telling your friend or secretary what your password is, and to fetch your messages and preferences. The same problem exists with things like the XP tracker from cowofdoom. Now I've met Will, and I'll go kick his ass if anything happens to my password (I still don't use it anymore), but the point remains is that you simply can't trust a random third party with your authentication credentials.

Another issue for full time syndicate use is that you can't guarantee that a particular hub would be syndicating the content correctly (ie: non-maliciously). There's nothing to stop them from redirecting your votes to their writeups or other things. Similarly, there is no safegaurd against people changing words inside of your messages or otherwise misreport what is available here.

dem bones says "Get the hell off my website."

You simply can't trust a downstream publisher. There is no secure end-to-end connection that can be guaranteed or authenticated in any way. Even open source websites don't have to report their source correctly. There's not really any way that you can compile the source yourself and run a "pure" binary. Before we ship any binary version of an E2 client on this website in an official capacity, I'll have to inspect the code myself thoroughly (or any other code administrator), and build it on one of our machines here. Trojaned binaries are a problem.

Some of E2's features, such as leaving softlinks, won't work if you are using a third party website to view it. Right there, I feel that that this is a great drawback. Sure there are technical ways around this that I can think of, but they involve a lot of pageloads to the e2 server, and break a lot of the caching that is done.

The temptation to want to reduce load from E2 is great. I personally would love to see pages load lightning fast without kicking all of your asses off ;). The problems that exist with non-first-party endeavors as such is the same as trust in any P2P application. Without some sort of crazy encryption or verifiable digital signatures, this sort of thing wouldn't work because of the security implications.

On the other hand, what this does make is some cool ways to keep your writing on E2, and your contributions here on another webpage in sync. Until we have some of the cool content management features due in later this year, there are some neat things that you should be able to do on a homepage with this sort of item. I can see wanting to collect certain writeups and format them differently, or want to perhaps make your own sort of collections of writeups. We as E2 are merely publishers (under a legal sense), and you're free to reproduce your writeups wherever you see fit. Syndication of things like softlinks and stuff are kinda cool (if not an excusable rip-off of our content here).

In the end, the point of E2 on one hand was to create a website where you could read for hours and days without leaving. Seeing as you've all been here for years, I'm assuming it's working. Spreading it across hubs is a neat novelty idea, and I definitely applaud its efforts. It's quite a cool trinket, but it isn't right for E2 full time.

Open for comments. Anything longer than two messages please send to jaybonci@everything2.com

Those funny characters in the node titles sure do get annoying - near impossible to search for (and actually impossible to type in some cases).

One possibility given no changes right now is to have two (or more) e2nodes for each of these. One of these nodes will be designed to be searchable, the other has the correct topic. A link and lock on the searchable one to the proper title.

The problem with this? Its a ton of work to find all of those nodes and make searchable nodeshells for them. Lets just say it would be difficult at best and constant work and maintenance to create proper titles for nodes.

There is another possibility (other than giving everyone a way to easily search for nodes that one can't type).

Currently, the title of an E2 node is a unique identifier within the database. Searches are done against this title. What if this title could be both correct, and searchable at the same time? In effect two titles.

What you say?! Its not as whack as it seems at first. When creating an E2 node, or updating the title of an E2 node the update will also create a 7 bit ASCII version that is hidden from all but search. This is near nil impact on server load. All searches will be done against this title. (Un)fortunately, this will not be a unique title - and in theory should be an indexed column.

When doing a search, the same translation is done upon the search words. This is slight hit. In theory, this search key will have a number of s/// preformed upon it. It would likely be best to have it be studyed first. Whatever the case, you get something like this:

sub deFunk()
{
  my ($words) = @_;
  study($words);
  $words =~ s/è/e/g;
  $words =~ s/È/e/g;
  $words =~ s/é/e/g;
  $words =~ s/É/e/g;
  $words =~ s/ê/e/g;
  $words =~ s/Ê/e/g;
  ...
  return $words;
}

Ok, you say - so we made some high ASCII translated to 7 bit ASCII. Big deal - mysql is supposed to be able to do that. Well, yes, it does for some of them. But it doesn't for all of them - most notably the ö which gives it major headaches. But wait - there is more. Not only can we solve the high ASCII problem, but we can solve world hunger the hyphen problem. To the search engine, 'foo-bar' is not the same as 'foo bar'. Adding the line $words =~ s/-/ /g; will make it so that searching for 'foo' or 'bar' will find 'foo-bar'.

Earlier I glossed over the unique thingy. When doing a search, if one and only one node comes back then it is displayed. If findings finds two 'exact' matches it should display the one that matches the text entered. If nothing matches the text entered, display a findings page as normal.

In this case, the challenge will not be maintaining two sets of nodes for all titles - one proper and one not; but rather have just the proper title. It won't matter how people link to it be it with the funny character or not. If there is one, and only one title that properly matches - it will be found.

In theory, one could also deFunk() html entities too. This would mean that there would be a few more matches to do (though you don't have to worry about case in this case). Remember, study adds a bit of overhead at the start but drastically cuts down on the cost of matches done against the string later. The advantage here is that if a person (wrongly) put a html entity in the html title without a pipelink, it would still find the correct node. One word of warning here is that nodes should never be created with html entities (at least in cases where the html entity is an otherwise 'normal' ASCII character). Allowing such would only create headaches down the road, though could be addressed by a deJazz() function to properly title the node title.

Update - I wrote a caching e2 proxy. If you're in edev, you have the URL (check your inbox)... If you're not but you have a strong desire to see it in its horribly incomplete state, /msg s_alanet. And those of you viewing this page through the proxy... Salud! More information below under "The Implementation."

Introduction

E2 has some problems. We've got a huge database to store, and hefty bandwidth requirements. We don't have the cash to buy a shiny cluster to host the site, or a bigger, phatter pipe to feed it through. There isn't really cash to hire full time developers or editors, though we have a lot of really wonderful volunteers that should be paid.

I was thinking about the whole distributing e2 thing, and I realized that we need a stopgap. An interim measure. Maybe a long term measure (after all, making a fully distributed e2 system is a big task - it might not be done for years!).

I was also thinking about the difference between e2 and a blog. I like my friend's blogs because they're very personal. I look at whose I want. If they're fancy I can have little discussion in the comment boards with my friends. I post stuff, they post stuff back - it's a lot of fun. But not very interesting for other people. E2 is the opposite - it's all about high quality writing for everyone to enjoy. If I post a story about my dog, it had better be good, not just some drivel that only people who have seen my dog will appreciate. Factual information - which would be boring on a blog - has a strong presence here.

We also have a lot of neat tools - the e2 node tracker, the E2 Nodegel Visualizer, and a lot of really cool clients.

How - I thought to myself - could these wonderful things be combined? It seems like they're all connected... but how?

And then it all came together.

The Idea

An Everything2 Hub - or an E2 Blog, or whatever you want to call it.

What I have in mind is a website that you can drop into your webspace on your ISP or university. It's a PERL or PHP script that automagically does everything to set itself up - a seed, so to speak. It probably uses flat files for maximum compatibility.

• It provides a blog syched with your daylogs. You post as you see fit - it deals with making sure the right daylog writeup on E2 is appended to, edited, or created. Sort of a blogger for the E2 set. It also displays the blog entries (including converting hardlinks to href's).
• It shows statistics relevant to you - latest writeups, for instance. People who are interested in you can hit your e2 hub and see what you've written. Optionally, more statistics could be available - most popular writeups, "goodest" writeups, least voted on writeups, etc.
• It provides a cached view of E2. Click on a link in the blog and it'll take you to a cached view of the node. You only hit e2's server if you want to.
• It provides "user services" to the owner of the blog. An integrated node tracker would be a good start (and pretty much a necessary feature to do the statistics). A nodegel visualizer would also be useful - in fact, there could be a whole subsection devoted to managing one's homenode. In addition, a scratchpad would be good (saving E2 from rendering lots of pageviews of the scratchpad), and the ability to make rough drafts of writeups, then upload them when they're ready, would be even better.

Basically what I'm talking about is a web-based client integrated with a web-based user info tool. A hub for your everything experience.

The Implementation

In the process of writing this thing, I've found some lingering XML malformations in E2's code. I've submitted patches for these. If you're using the e2hub and you get a blank page, it's probably because you hit a page with an unescaped character, like a bare & or something.

I also went through and cleaned a lot of stuff up in the code... It caches user information, nodes, and displays them. I'm about ready to call it 1.0 - I have a few issues with Simpleton's wonderful E2Interface library, mostly relating to me using it in ways it was not designed to be used, and when I clear those up, I'll be posting the code to the hub somewhere.

Last updated 15:52 23/2

So I've actually started to code some of the stuff mentioned here. Specifically, what I have at this point is a web-based caching E2 proxy. In other words, it grabs data from the various XML sources in the site to generate an e2-look-alike. It's got a 100 megs of cache space, and it keeps data cached for one hour. This is all tweakable. Right now it supports viewing nodes and users. If you try to view something else it will tell you to go back to E2 and look at it. The whole thing is templated.

This is the key to the hub concept, in my mind. Without a cache - without an interface to E2 - then I'm just talking about fancy newsletters. Now that I have a pretty solid cache implementation, I can really go somewhere.

If you have any questions, feel free to contact me.

Extensions

It's also not hard to imagine such a tool being turned towards making E2 more approachable. How often have you heard, "It seems nice, but I just don't think I'd fit in..."? We are losing talented contributors every day because they don't know what to make of hot nude thespians. Sure, it's a part of E2's charm.... but we're reaching the end of where charm can take us. We need quality to reach the next stage of development.

How better to attract quality contributors for music writeups than a hub that focuses on, say, "Music Reviews"? Or a "Computer Science" hub that focuses on terminology and concepts? A "classical literature" hub? E2 is a big place. If we want to lure in experts and promote really quality writing, we're going to have to start filtering things to fit the audience. Metanodes are a good start, but they're not enough.

It should always be easy to start free-associating - clicking through the database, wherever the information flow takes you. But if we want to attract serious contributors, we need to be able to present them focused information.

Conclusion

There are a lot of other areas that this concept could be extended along - letting people check their messages from their hub, for instance, or making a hub "farm", sort of like Sourceforge does for software projects or LiveJournal does for blogs. Such a site could even charge people a small fee for advanced functionality, or faster service - thus helping E2 stay afloat.

Call for Comment

But of course, all this is just conjecture. I would like to make it, or something like it, a reality. E2 should live on! So please, if you have any comments, thoughts, or suggestions on this, drop s_alanet a message or e-mail me at <bgarney@purdue.edu>. I'll note them down here (as well as thanking you profusely for your willingness to tolerate my ramblings :). If you are struck by the urge to start coding this, talk to me, too... I'd like to help (I don't have time to write it on my own).

---

And the responses come in...

Ouroboros says re Everything2 Hubs: like a "distributed" E2?

you said "re hubs: Except here we have everyone talking to a central server. On Making E2 Distributed, I was talking about a fully distributed, peer to peer model. This is just a bit more hierarchical."

(That is to say, this is an extension to the existing architecture whereas distributed e2 in my mind is a total redesign.)

PhillC sez re: Hubs. I like the idea, but perhaps the easiest 4 hubs would be E2 People, E2 Places, E2 Ideas, E2 Things. They could all be easily pulled from the DB.

you said "re hubs: True... Though given the arbitrary nature of the people/place/idea/thing distinction, I think topical hubs would be more useful."

HongPong sez:

well i think it would be good to get better clients available for everyone. there is no good reason for scratch pads. however i think it will be more effective to look at where e2 is getting processor eaten. also i think it would be good to have a perl/PHP script which could generate fuzzy connections from your nodes to other topical nodes... within topics and those of your friends...

i kno from my metanodes that e2 has a broad-level anarchy to its organization. my general idea is to make a new 'metanode' node type which appears bold in the softlinks.. ie my Middle east would always appear bold. this would point people to topical indices... eventually fuzzy math + something like voting could be used to create topic fields. think fractal organization

have you seen this thing called 'helloworld' which works by making fuzzy collections of data. http://www.cooperatingsystems.com/helloworld/factsheet/index.html

i support anything making topical stuff and metanodes better. i think in particular it could be possible to flag certain nodes and have it generate others you might be interested in... but how to do this without crushing the server...

me sez:

"re hubs: A goal of the E2 hub is to offload things from the central server. I think that a generic mechanism to 'cache' unfinished writeups would serve nicely in that role, as it would give people a chance to collaboratively review or individually rewrite wus before posting them on the live server. The scratchpad is a very convenient part of e2, but it's also a source of database usage and system load.

A huge portion of e2's load is caused by the complexity of its pages. Every pageview results in dozens and dozens of comples queries, as well as in the execution of lots of code. Making the system more complex by adding filtering rules or nodetypes will do nothing to reduce pageviews caused by casual browsing, while making page renders more expensive. Quite the opposite of what we wish to accomplish!

On the other hand, if we write a hub that works to off-load as much as possible - both caching content for casual browsers, as well as providing a "draft then upload" paradigm for nodes in progress, we cut two expensive operations out of the loop - we reduce page renders for passive consumers by at least an order of magnitude, and page renders/database updates caused by writing/rewriting.

As of 1pm PST today, E2 has gotten fifty thousand hits. What if we could reduce that to 25k?

(Addendum to that: helloworld does look pretty neat... but for E2, I think that a web based solution will be more amenable than a program to download and run.)

A quick note about how E2 works. Everything tangible to the Everything engine is a node. Users are nodes. Writeups are nodes. The things that we normally think of as 'nodes' are in fact 'e2nodes'; which are, of course, a type of node. Types of nodes, in fact, are themselves nodes. I think you're getting the picture by now: everything in Everything is a node. I'm a node, you're a node (if you've got an account here), nate's a node, Guest User is a node.

Nodes have their basic information stored in the node dbtable. They're identified by a completely unique identifier, the 'node_id' field of the node, and column in the database.

All very well and good. We have nodes. We can identify them by their node_id. Things like ownership of writeups, messages and membership of groups are all dictated by the node_id of the user's account, because that's the definitive way of identifying a node.

And this is what gets us into trouble.

In ecore, there's a handy dandy little function called updateNodeData. This function will manipulate fields of a node owned by the requesting user, in response to certain CGI parameters, which I'm not about to name, passed to the index.pl script. By default, it allows a user to manipulate any field of the node. This would, naturally, be quite a large security hole: you could, for example, change the owner of your bad writeups to someone else. You could change your username to be more or less anything you like. So to avoid this, sensitive fields aren't allowed to be written. These fields are named in a setting node, and access to any of these fields will be denied to anyone except members of gods. So, all is good. Right...?

(It's important to note, at this point, that the updateNodeData in the ecore used by Everything2 is very different from that used in the publicly available ecore's, so reasoning about one's behaviour from the other's is not exactly safe...)

One boring Friday night, I was poking around idly checking the sanity of things to make sure that things that shouldn't be writable weren't writable. Someone suggested that I check the node_id. The idea was ridiculous: you can't change the node_id of a node, it's unique. Even if the node_id field wasn't protected by the conventional mechanism, the database would refuse to update a node entry such that it violated the uniqueness constraint on node_id (the primary key of that table). It was inconceivable that it would work.

Logging in as a spare account, I entered the relevant runes to change the node_id of the account node to that of Guest User. If anything at all were to happen, I reasoned, that would at least be entirely harmless. The worst that could happen would be to clone Guest User into the old account node, so choosing the most harmless user account possible seemed a reasonable precaution.

I hit enter. Nothing seemed to happen. I checked the node_id of the homenode of the account... mysteriously, it seemed to have actually changed the node_id. Scratching my head, wondering how this had happened, I logged out... to no apparent effect. Yes, my theme and nodelets were still there, but the epicentre looked a lot like Guest User's.

Uh oh.

At about this time it started to occur to me what actually happened. The updateNodeData function didn't do exactly as I'd expected. It doesn't do an atomic database update. It takes the node data loaded from the database at the start of the pageload, and commits it to the database. And the only way it can work out where to put the node data is the node_id field in the copy of the node. Which we've just managed to mutate into Guest User's node_id. At no point was a duplicate node_id created, the new node's data was simply written over the data for the Guest User node. Pageloads for anyone not logged in to E2 are taken via the Guest user account. Which is identified via its node_id, so to all intents and purposes, huggles and Guest User were one.

Aha.

The obvious course of action was to create a new Guest User account to replace the old one and put things back pretty much the way they were. This duly done, I contacted an edev god with the intention of letting them know what was happening. Unfortunately, by this time, and before I'd adequately explained how I was able to more or less patch things up and how this wouldn't work for any other node, my accomplice had already pulled the same trick with nate's account.

At about this time, JayBonci appeared online, and I started explaining things to him; he promptly pulled the web servers down and started working out what had happened, and checking nothing nefarious had been done by the guest user account while it was masquerading as someone else.

After a brief amount of faffing around with IRC clients, we logged on to #everything and started trying to contact gods, begging forgiveness, and generally make a nuisance of ourselves. Nate duly appeared and forgave us, recognising our intentions as basically benign and realising the potential impact if the security hole had been discovered by more nefarious parties.

In summary, then: Jay is unquestionably the man, nate is about the coolest guy since Frosty the Snowman, and I couldn't be more regretful if I were Ronnie Corbett.

Oops.

---

As a note to Jay's root log, the Guest User node isn't fully restored. As a memento of the event, it's still noted as created on October 12th.

This is edev's collaboration node, essentially for discussing general crap without having to clutter up the usergroup too much. See usergroup message archive for our past discussions on the usergroup (I'm afraid that the old, imported writeups are all timestamped at import time and are likely to be out of order). Just add your comments below the existing ones, along with your username. Add new discussion topics above existing ones. Please remember that edev is meant to be a *technical* discussion group. e2 policy should not be discussed here.

(only edev, editors and gods can view/edit this node. click the 'edit' link above to lock the node for editing, edit as necessary, then click 'unlock' when you're done, so that other people can edit it.)

Other available methods of discussion include the usergroup, the sourceforge bugtracker/featurerequest stuff (http://sf.net/projects/everything2) and #edev.

I've tried to summarise recent usergroup messages/discussions here - fuzzie

timestamp format unification, localization (and customization)

(da-x/)
There are currently 4 different timestamp formats shown in E2:

• Writeup header: 'Sun Oct 01 2000 at 19:14:32' (wrong shift fixed)
• User search: '2002-10-25 16:43:20' (fixed, now localized and matches the writeup header timestamp format)
• Inbox messages: '2002.11.07@20:05' (fixed, now localized)
• Server time: 'Thursday, November 7, 2002 at 20:30:18'

On the short run: Very confusing IMHO. Currently only the writeup header is localized (according to to the user's time zone). Writeup headers use the parsetimestamp code. If we use parsetimestamp in all the other areas we can get this issue unified.

On the long run, prehaps we should let the user pick up her preferred timestamp format, like in slashdot. I prefer Stardates.
(/da-x)

sherlock 3 e2 plug-in

With the advent of Jaguar (Mac OS 10.2.x), a new version of Sherlock came along. Now, I won't bore you with the details (http://apple.com/macosx), but basically it would be possible with the new version of Sherlock to make a sort of read-only clientdev viewer, much like the eBay plug-in already in use in Sherlock 3.

I am researching ways to make Sherlock 3 plug-ins, but help would be greatly accepted.

On a similar note, AFAIK old Sherlock plug-ins no longer work. I will research this further. - TheChronicler

special superdocs

A suggestion for a special setting for such nodes as everything faq, node heaven, etc, which basically means that we either don't allow the creation of e2nodes (nodeshells) with that name, or we jump to the superdocs by default and provide 'see also' links (or something similar) back to the nodeshells. - fuzzie

lj-cut tag

As seen on livejournal, this is basically a structured way to handle a summary of a writeup; the idea being you'd enclose the main portion of your writeup with a certain tag which means that for users with a certain user preference set, it wouldn't appear until they clicked a link to display the whole writeup. This would mean only the summary would appear. It would also mean that people could hide solutions to problems, answers to jokes, etc, without being accused of cluttering the database with multiple writeups.

Another way to do this might be to surround the 'summary' bit of your writeup with a tag, and that is the only bit which appears for users viewing an entire e2node with a certain pref set.

It has been pointed out that a big problem with this is the existing base of writeups, and that people might not bother with it; but I feel it'd be useful anyway - fuzzie

preventing duplicate messages

There has been a lot of discussion on this topic recently. 'ascorbic says There's plenty of ways it could be done (eg. incrementing value in a hidden field, with a unique index in the db). The question is if it's really worth it. it's not like multiple messages are a major crisis.'

scratchpads

drinkypoo says multiple scratchpads would solve the browser interference problem and also make life easier in a bunch of other ways. I like the idea of scratchpads which can be restricted to users or groups, too.

Also, scratchpads are currently limited to 64k, but the truncation isn't obvious upon submission. The 64k limit is a database one, but could perhaps be worked around by using multiple records. - fuzzie

As I've had to prove, scratchpads are of more or less unlimited size, and have been since february: Just when scratch pads couldn't get any better.... Some people seem to see limitations though, and I suspect this is a browser issue, but don't have enough data to back that up... call

we might also consider the idea of just commpressing thje text, you can compress palin text really really well, for not too much cpu time. like at some point we shoudl have a real system with multiple records, but for now... --Delta-Sys

Extra scratchpads as a level bonus might not be a shitty idea. - Unless

drinkypoo: Scratchpads restricted to/editable by specific sets of users/groups is served by collaborations .. it'd be really nice if creating those could be a (lowish? 5?) level power, with the ability to nuke them and have X amount of them at a time, dependant on level? - fuzzie

I think it's a really silly idea to make extra scratch pads a level bonus. scratch pads are a useful tool, I think we shoudl restrice futures on a per-level basis only for fun stuff (who created a nodeshell, etc) and judgment, trust issuese(number of votes, C!ing, etc) and not for useful, potentally nond-impruving stuff (spellcheck(yeah, right) extra scratchpads, etc)...not EVERYTHING has to be some sort of high-level bragging right. --Delta-Sys

Extra scratchpads *are* fun stuff. Very, very few users are having trouble with running out of space on their scratchpads. For the vast majority of noders, another scratchpad will be another place to put random shit. Like an extra homenode, basically (which is what the first scratchpad is to many users). -- Unless

misc feature requests

Please delete feature requests after they've been implemented.

Hey, wouldn't it be neat if we had some sort of user 'last seen' time on private messages? -- call ... showmessages (patch), user settings (patch)

In addition, if it were possible to archive messages from the chatterbox nodelet (without a a potentially lengthy visit to Message Inbox) that'd be nifty. We needn't provide an extra checkbox next to the messages -- archiving happens less frequently than deletions: it could appear (possibly only as an option) as something like a single-letter URL link like (r)... say, (a)...? -- call

Archiving isn't less frequent for everyone, I archive a lot of stuff. In fact, I'd find it useful if I could switch the functionality of those checkboxes from deletion to archiving. - Unless

The only way to get things done around here seems to be to do 'em on the client side. So I shall whip up a bookmarklet to archive messages ticked in the catbox rather than deleting them. Howzat? Done: see E2 Bookmarklets. -- call

stuff which keeps coming up:

The 'edevify!' link in your Epicenter is used to add the node you're viewing to the weblog (displayed writeups) on the edev usergroup page. Don't use it unless you're adding an edev-related writeup or edevdoc you think we should read.

Yes, sometimes the + and - votes on a writeup don't add up properly to the reputation given. It happens.

Yes, Java Chatterbox is currently broken.

The addition of new writeuptypes is not an edev issue, it's something for the admins to decide.

If you get bestowed votes after you've spent all yours existing ones, when you use them up, you get your experience bonus for spending all votes on your level again. Again, it happens.

Autovoting scripts and scripts to rearrange the softlink order are not allowed, and will likely be banned if spotted. Please be sensible and put sleep() statements in your scripts. See clientdev: new xml ticker output and everything data pages for help with coding scripts which need to access e2.