QR-Code pointing to DLTJ

Tom emphasized the need to have an activity that is relevant to the technology. As he put it, “Use the technology to ampliy the activity.” In this specific case, the 2-D barcodes pointed to text, pictures, and videos that provide additional background to the components depicted in the World War II Memorial. As participants mentioned in the video, it is a way add context to the experience of walking through the memorial.

I had one minor quibble with the execution of the project. The presenters were using EZcodes, a 2-D barcode format licensed exclusively to ScanBuy rather than the emerging de facto QR codes. The problem with EZcodes is that what is encoded is an identifier that is translated by the ScanLife smartphone app to the final destination. By contrast, a QR code has the actual content — a short snippet of text, a URL, a phone number, etc. — actually encoded in the barcode. With an EZcode, the application on one’s smartphone has to look up the value of the identifier at ScanLife’s service before going to the final destination. With a QR code, the smartphone application can go right to the destination website.

The EZcode/Scanbuy scheme has privacy and sustainability issues. First, according to the terms-of-service for the ScanLife reader, each reader application is assigned a unique identifier; because the application must contact the ScanLife with the 2-D barcode identifier to find the value behind the identifier, ScanLife knows everything you scan. Secondly, the ScanLife server is a mandatory intermediary in the process, so if ScanLife goes away all of the barcodes become worthless. This is somewhat similar to the problem with music encumbered with digital rights management; if the server is unavailable, the music file is worthless.¹ QR codes, though, since the data is encoded in the barcode itself, does not have either of these problems.

Footnotes

1. See, for example, DRM sucks redux: Microsoft to nuke MSN Music DRM keys, and also how Wal-Mart Reverses Decision To Shutdown Digital Music DRM Servers.

Post from: Disruptive Library Technology Jester

Experiential Learning Enhanced with 2-D Barcodes

This year the ALCTS Forum at ALA Midwinter brought together three perspectives on massaging bibliographic data of various sorts in ways that use MARC, but where MARC is not the end goal. What do you get when you swirl MARC, ONIX, and various other formats of metadata in a big pot? Three projects: ONIX Enrichment at OCLC, the Open Library Project, and Google Book Search metadata.

Below is a summary of how these three projects are messin’ with metadata, as told by the Forum panelists. I also recommend reading Eric Hellman’s Google Exposes Book Metadata Privates at ALA Forum for his recollection and views of the same meeting.

ONIX Enrichment at OCLC

Renee Register, Global Product Manager for OCLC Cataloging and Metadata Services, was the first to present on the panel. Her talk looked at a new and evolving product at OCLC on the enhancement of ONIX records with WorldCat records, and vice versa. ¹

As libraries, Renee said “our instincts are collaborative” but “our data and workflow silos encourage redundancy and inhibit interoperability.” Beyond the obvious differences in metadata formats, the workflows of libraries differ dramatically from other metadata providers and consumers. In libraries (with the exception of CIP and brief on-order records) the major work of bibliographic production is performed at the end of the publication cycle and ends with the receipt of the published item. In the publisher supply chain, bibliographic data evolves over time, usually beginning months before publication and continuing to grow for months and years (sales information, etc.) after publication. Renee had a graphic showing the current flow of metadata around the broader bibliographic universe that highlighted the isolation of library activity relative to publisher, wholesaler, and retailer activity.

Diagram of the Process of Enhancing ONIX Records, from OCLC Services for the Publisher Supply Chain Webinar, August 2009

In her experience, Renee said that libraries need ways to enable our metadata to evolve over time and allow for publisher-created metadata to merge effectively with library-created metadata. The bibliographic record needs to be a “living, growing” thing throughout the lifecycle of a title and beyond. In concluding her remarks, she offered several resources to explore for further information: the OCLC/NISO study on Streamlining Book Metadata Workflow, the U.K. Research Information Network report on Creating Catalogues: Bibliographic Records in a Networked World, the Library of Congress Study of the North American MARC Records Marketplace, the Library of Congress CIP/ONIX Pilot Project, and the OCLC Publisher Supply Chain Website.

From MARC to Wiki with Open Library

Karen said right at the start that the Open Library project is different from most of what happens in libraries — it is “someone outside the library world making use of library data” — although the goal is arguably the same as others — “One web page for every book ever published.” As such, the Open Library isn’t a library catalog as librarians think of it in that it is not a representation of a libraries inventory. It has metadata for every book it can know about and a pointer to places where the book can be found, including all of the electronic books in Internet Archive (Open Content Alliance, Google Public Domain, etc.) as well as pointers back to OCLC WorldCat. Karen’s role for the project is that of “Library Data Informant.” The Internet Archive decided that they needed someone who understood library data in order to try to use it. From Karen’s perspective, she is trying to be a resource for project but not give them any guidance on how to implement the service. She is curious to see what the project would do when bibliographic data is viewed from a non-librarian perspective. If they have questions, or if they have assumptions about data that are wrong, then she intervenes.

Karen went on to briefly describe the Open Library system. Open Library doesn’t have records; rather, it has field types and data properties. In this way, it uses semantic web concepts. “Author” is a type, “Author birthdate” is another type, and so forth. There are no set field types, so if the project gets data from source for which a type doesn’t yet exist, it can create a new one. Each type can have data properties such as string, boolean, text, link, etc. Nothing is required and everything is repeatable. Everything — types, properties, and values — gets a URI (a URI is an identifier like a URL, but conceptually a superset of the universe of URLs). Titles, authors, subjects, author birthdates, and so on have URIs. Lastly, the underlying data structures are based on wiki principles: all edits are saved and viewable, anyone can edit any value, anyone can add new types or properties, anyone can develop their own displays, etc.

The data that is now in Open Library came from a variety of sources. They started with a copy of books from the Library of Congress, and continue to receive the weekly updates. They performed a crawl of Amazon’s book data. They have gotten some from publishers, libraries, and individual users. The last is perhaps the most interesting because it is mainly people outside the western world who are otherwise having trouble getting their works recognized.

Problems, Issues, Challenges, and Opportunities with the Data

Names -

“These library forms of names? Honestly no one but us can stand them.” Even something as simple as the form of last-name-comma-first-name is troublesome. No one else uses this form of the name: Amazon, Wikipedia, etc. In processing these, any information between parenthesis has been deleted, birth and death dates move into separate field types.

Titles -

In working with the Open Library developers, this is one place that Karen tried insisting on applying a library practice: knowing the initial article. For us, this is important for sorting books in alphabetical order. The developer response — why do we have to sort in alphabetical order? “Where else but library catalogs to we see things sorted in alphabetical order? Not in Google, not in Amazon, not anywhere. Alphabetical order is not in the mindset anymore.” They also found that the title might include extraneous data. Amazon, for instance, appends the series title in parenthesis to the main title. This is a demonstration of how other communities are not as concerned about strongly typing and separating information into fields. Amazon, of course, has reasons for series information into the main title: it helps sell books.

Product dimensions -

Publishers and distributors need to know characteristics of an item such as height, width, depth, and weight; they, of course, need to put it in a box and ship it. Libraries, concerned about placing the item on the shelf, record just height. Recording pagination is different, too: libraries use odd notations “ill. (some col)” and “xv, 200p.” versus simply “200 pages.”

Birthdates -

Librarians use birthdates to distinguish names; if there is no need to distinguish a name, birth and death dates are not added. Someone looking at this from the outside would ask ‘Why don’t all authors have birth and death dates?’ This can be useful information for viewing the context of an item, not just to distinguish author names. Open Library ran author names against Wikipedia to pick up not only birth and death years, but also the actual dates.

Subject headings -

Open Library using Library of Congress Subject Headings was out of the question. In processing the data, the Open Library developers just broke them apart into segments and used them. But because they were able to do data mining on the subject field types, they did find statistical relationships between the disassembled precoordinated headings and were able to present those to the user.

The View of the Data -

Rather than a traditional library view of long lists of author-title, the Open Library (in its next version coming in February) will have several different views into the mass of data: Authors; Books (what we would call FRBR ‘manifestations’); Works; Subjects; and eventually places, publishers, etc. For example, when searching for an author one would get the author page. On it would be all of the works from the author as well as other biographical information. It looks similar to a WorldCat identities page, except it is the actual user interface built into the system. Similarly, every work will have a page, and at the bottom of it one will see all of the editions of the work. Also, each subject will have a page, and one will see a list of works with that subject as well as authors who write on that subject. As Karen said, “The subject itself becomes an object of interest in the database, not just something that is just tacked on to the bottom of the library record.”

Data mining -

With the data in this format, it is possible to perform data mining actions against it. For instance, simple data mining such as country of publication, popular places that appear, etc. When they had the problem of author names — knowing when to reverse surname and forname — they ran the names against Amazon and Wikipedia and retained the ones where they found the order of the entry was the same. The Open Library developers are also experimenting with data mining to find publisher names. Publisher names, of course, vary dramatically, but by using ISBN prefixes they can pull together related items into a “publisher” view.

Karen suggested watching the Edward Betts’s site, one of the developers of the Open Library project with an eye on the data mining aspects. She said it is fun to look at our data when it can be viewed from this different point-of-view. She also said to watch out for a new version of the Open Library website coming in February.

Google Book Search Metadata

On the input side, Google is getting bibliographic metadata from over 100 sources in a variety of formats. MARC records are coming from libraries, union catalogs, commercial providers (OCLC), publishers/retails (one publisher supplies records in MARC format). Google also gets ONIX records from commercial providers (such as Ingram and Bowker), publishers, and retailers. Google is especially interested in data from non-U.S. retailers because it is a source of information about books published outside the United States; it helps facilitate discovery of items that they may not otherwise encounter in the publisher and library programs. Google also receives records in a variety of “idiosyncratic formats” — for example, publisher-contributed metadata (via the Publisher Partner Program); information associating books with jacket images; name authority records (from LC); reviews; popularity signals (sales data as well as anonymized circulation data from some library partners, useful for feeding into the relevancy ranking algorithm); and internally-generated metadata (for instance, whether a book is commercially available or not). Google processes all of this information to come up with a single record that describes a book. At this point they have over 800 million bibliographic records and one trillion bits of information in those records.

All of these records from all of these sources are processed and remixed with Google’s parsing algorithms about twice a week. The first step is to transform the incoming records into a “less verbose format” for storage and processing. It is a SQL-like structure that allows elements of the metadata to be queried. Records are then parsed to extract specific bits of information, transform the bits as necessary, and write the information to an internal “resolved records” data structure (a subset of the data coming from the input formats). In the presentation, Kurt had examples of how making inferences from data coming from both MARC and ONIX can be troublesome. Parsing also involves extracting “bibkeys” from the records to aid in matching across sources of data. Four types of identifiers are extracted from bibliographic records: OCLC numbers, LCCNs, ISBNs, and ISSNs. They provide usually useful signals when matching bibliographic and help with assertions that two records describe the same manifestation. Google also tries to parse item data when present in records representing multi-volume works, enumeration and chronology. They will also treat barcode as a form of a “bibkey” if they get it from a library. The parsing algorithm will also split records containing multiple ISBNs representing different product forms (e.g. hardback, paperback, etc.).

With all of this data parsed into records, Google starts its clustering process where records are examined and attached to each other. Bibkeys provide significant evidence for relating records to each other, but bibkeys are not always present in a record (non-U.S. records and older records frequently contain no bibkeys). The algorithms then fall back on text similarity matching using title, subtitle, contributor and other fields such as publisher and publication year. The results are clusters of records representing the same manifestation. An algorithm then attempts to derive the “best-of” record for a single cluster from all of the parsed input records. This is done in a field-by-field voting process based on the trustworthiness of individual fields from record sources.

Kurt went into some of the challenges facing the team building the clustering and best-of record creation algorithms. For instance, in dealing with multivolume works they know of 5 numbering schemas with 3 number types in 15 different languages. Enumeration is now showing in the public display, but the development team is still working with unparsable item data due to inconsistent cataloging practices between institutions…and sometimes inconsistencies within an institution. Another problem is non-unique identifiers. In the current data set ISBN 7899964709 is shared by 75 books and ISBN 7533305353 is associated with 1413 books. There are also poor quality or “junk records”. Kurt said his favorite was “The Mosaic Navigator” by Sigmund Freud published in 1939. These are hard to identify with an algorithm, and they rely on reports of problems that enable the developers to go in and “kill” the troublesome record. Another example is a book by Virginia Woolf where the incoming record had conflicting information; it had two 260 fields that contained different dates (1961, correct, and 1900) with fixed field information that strongly suggested that 1900 was the single date of publication. When the data problem is systematic, they can identify it and compensate for it. Kurt’s example for this case was “The United States Since 1945″ published in 1899. This one was highlighted in Geoffrey Nunberg’s criticism of Google Books metadata. In this case, there was a source of metadata from Brazil that when they didn’t know the date of publication would use 1899. When Google went back and looked at the date distribution of books there was a huge spike in 1899. Once Google knew about it they were able to go in and kill that information from that source of records. ³

In closing, Kurt said that Google is committed to engaging with the library community on improving metadata and metadata processing.

Footnotes

1. For those not familiar with ONIX, it is a suite of standards promulgated by EDItEUR for the interchange of information on books and serial publications. It is primarily used as the communication channel between the publishing industry through distribution chains to retail establishments.
2. By the way, it seems like BISAC is an acronym for “Book Industry Systems Advisory Committee”, the former name of the Book Industry Study Group.
3. A side note: Google isn’t the only one tripped up by this. If one searches for the ISBN of the item, 0195038487, you get to more than one site that has the same incorrect publication date. At least Google is attempting to clean up the data!

Post from: Disruptive Library Technology Jester

Mashups of Bibliographic Data: A Report of the ALCTS Midwinter Forum

Jay Jordan’s remarks during the OCLC Update Breakfast and the discussion at the Developers Network table at that breakfast generated further fuel for my previous philosophical thoughts on “Who is a member of the OCLC Cooperative?” In the context of things like Developer Network API keys¹ this question of who is a member of OCLC the cooperative and who is not meets the on-or-off, ones-and-zeros nature of computers. One can’t “kinda” have an API Key unless that capability is programmed into the software (or a human chooses to override the established rules for who has a key).

The discussion around the table after Jay’s remarks tested some of the “edge cases” to the established hard-and fast rules. OCLC Governance, at least as I understand it, surrounds institutions who are members of the OCLC Cooperative. By implication, the benefits (and also the responsibilities) of membership in the OCLC Cooperative transfer to staff employed at those member institutions. People who work for governing (institutional) members of OCLC can get Developer Network API keys to create services for their library. And except for the fuzzy notion explored previously whether individuals are members of the OCLC cooperative, this all seems pretty clear.

But what about individuals in the employment of a library that is not a member of OCLC that want to create applications that benefit library users. Further, following Jay Jordan’s definition of partners OCLC wants to work with (as expressed during the OCLC Record Use Policy Council Update), that non-institutional-member individual can return value back to the cooperative (by contributing the ideas to the Developer’s Network Showcase, by contributing working code to a software repository, etc.). Isn’t that returning value to the cooperative? Should this person be granted a Developer Network API key even though their institution is not a member? Can we conceive of a process and guidelines by which this could happen? (Has it perhaps already happened in the human-created fuzz of OCLC-the-stewards overriding the established rules?)

I’ll state here that I would support the creating of a process by which anyone — from a member library or a consultant doing work on behalf of a member library or a 12-year-old kid) can request a Developer Network API key. What is needed is a clear understanding that those who get such keys gain some kind of adjunct membership in the OCLC-the-cooperative, and as so are expected to return value to the cooperative through their work with the API key. (There would need to be transparency from OCLC-the-steward on who in this category is getting keys, who is not, and for what reasons — at least in summary.) The area of consultants or other agents doing work on behalf of a library is an area that would need further exploration. I would offer that it is okay for one such agent to use an API key to help a particular client, but I also agree with Jay that it is not okay for that same agent to use that key to derive revenue from helping non-OCLC members with the same key. The latter fails the renumeration-back-to-the-cooperative test (either financial or intangible benefit) that Jay remarked on during Saturday’s Record Use Council session and one that the Record Use Council seems to be struggling with.

These are tough but interesting issues. We’ve left the days where the benefits of membership were tied to the delivery of shelf-ready cards and dedicated terminals for online original and copy cataloging. OCLC seems to be transforming itself from a library-used service to a world-used service. I give OCLC-the-stewards credit for listening to OCLC-the-membership about appropriate ways to make use of the shared resource that is WorldCat, and credit to OCLC-the-membership for pushing OCLC-the-stewards into creating ways to expand access to that shared resource.

Footnotes

1. “API” is an acronym for Application Programming Interface. In summary, an API is the set of rules by which one program can task another program for data or to perform a service. An “API key” is the mechanism through which a requesting application establishes the right to be able to use data and services of another application. In this context, the holder of an OCLC Developer Network API key can access the wealth of data and services being offered by OCLC.

Post from: Disruptive Library Technology Jester

More on What Does It Mean to Be a Member of OCLC

I think it is a statistical anomaly that many of the meetings I attended during ALA Midwinter were somehow related to OCLC. That statistical anomaly has certainly played out in postings here on DLTJ of my impressions of Midwinter meetings. Continuing with this thread of OCLC events, I attended the OCLC Update Breakfast Sunday morning for a membership-dues-paid croissant and orange juice, and to listen to Jay Jordon’s biannual update on the past, present and future of OCLC. What follows are highlights that I found interesting in the course of his remarks, but certainly not a comprehensive report of what was said. Video of Jay’s remarks where recorded and are to be posted at some point on the OCLC website (roughly six to eight weeks from now, if my memory of past events can be any guide).

WorldCat Growth since 1998

New “Search Engines”

OCLC Services in the Cloud

OCLC is also looking to help with collection management as a cloud-available tool. Working with the New York University Libraries, OCLC is bringing analytics to bear on collection management and space allocation decisions by helping with data about the location of items in the campus library, in the library’s “ReCAP” remote storage, and what is available digitally in HathiTrust. And speaking of HathiTrust, the public interface to 7.5 million volumes digitized largely through the Google Book Search partnership, OCLC is working with project participants to aid the metadata description of items in HathiTrust, to ensure that items in HathiTrust have records in WorldCat, and to add WorldCat Local as an interface to the HathiTrust collection.

Recent problems for Cataloging Partners

Post from: Disruptive Library Technology Jester

Interesting Bits from the OCLC Update Breakfast

On Saturday morning of ALA Midwinter 2010, Dr. Jennifer Younger moderated a session on the progress of the OCLC Record Use Policy Council. The meeting started with an introduction to the reasons behind the creation of the Record Use Council, the charge of the Council from the board of trustees, and how the framing of the discussion of the policy is guided by the values and history of OCLC the cooperative. There wasn’t much new here for those that have been following the progress of the policy discussion, so I am skipping over it most of it with the exception of a few notable topics. After that, I’m focusing on the lengthy question and answer session that followed Dr. Younger’s background presentation.

Highlights of the Background Presentation

WorldCat itself is now made up of 170 million bibliographic records and 1.5 billion statements of holdings from libraries. A policy is needed to create a viable business plan for sustaining this resource.

What the policy will cover: rights and responsibilities of members that have created WorldCat — the rights of members to use elements of WorldCat and the shared responsibilities to the members of the cooperative that go along with the rights; identifying acceptable use by third parties; what are OCLC’s rights to use the records on behalf of the members; and a process for collective participation in reviewing and modifying the policy over time. It will also have a “rather robust” preamble that answers the question of why a policy is needed, what problem is the policy is trying to solve, and what it is about WorldCat that necessitates a policy.

An Aside: What’s In a Name — OCLC-the-membership and OCLC-the-stewards

The View from the Database Level

1. As a supply of bibliographic records.
2. The ability to represent library holdings — the collective collection of libraries and the capability to reveal what libraries have in places like Google Book Search.
3. Knowledge organization pieces: taking the shared contribution of libraries and makes something more from it using authority control, terminologies, the Dewey Decimal classification system, FRBR work sets, etc.

It was interesting to note a non-U.S. perspective that the council has heard regarding the value of WorldCat. While most North American libraries strongly value WorldCat as a supply of bibliographic records (copy cataloging), the national libraries outside of North America are joining because adding their records to WorldCat gives greater visibility to their holdings. So the second and third value propositions above carry more weight than the first, which is arguably the most valuable aspect for North American libraries.

The challenge the Council said it is facing is to put enough controls in place to protect the value and viability of WorldCat while allowing enough flexibility for members, non-members, and OCLC to experiment and derive new, valuable services. One of the questions the review council is grappling with is how can the Cooperative use “community norms” to ensure the responsibilities assigned to the members are followed so we govern ourselves.

In taking this database-wide view, the council has set aside issues of individual record ownership and copyright of data in records and focused on what is valuable about the collection of records as a whole to the membership. WorldCat as a whole collective is copyrighted. As explained in the follow-up discussion with members of the council, the intellectual property law surrounding WorldCat records extends across many juristictions, so the council chose to focus at the database level.

Third Parties

In licensing WorldCat data to others, OCLC-the-steward is looking for remuneration of some sort for OCLC-the-cooperative. If the business use by a non-member third party is one that will harm the value and viability of the WorldCat Network, then the policy council wants to see it governed in some way. Remuneration can be in monetary form, where that external party pays a fee for the data. Or it can be in a non-monetary form, such as the quid pro quo with the internet search engines that have WorldCat data and in return drive traffic back to local libraries through linkage on WorldCat.org. As Jay Jordan put it sucinctly, “I’ll do a contract with anyone that returns value to the cooperative. Oftentimes, that is not cash.” It was stated that there have been attempts to download the entire WorldCat database. In order to be able to legally stop that, there must be a policy in place that prohibits it.

WorldCat as Linked Data

Questions

1. In Jennifer’s introduction, she talked about not only the value of the bibliographic records but also the value of the holdings. Has the Council looked a differing policies for bibliographic information versus holdings?
2. The discussion of linked data was incomplete due to time constraints. Has there been a discussion about a differentiation of value for different types or views of data? Machine access version human-oriented access? Linked data of some portion of the bibliographic record? Is the representation of the benefit of the world in general being taken into account in drafting policy guidelines?

Post from: Disruptive Library Technology Jester

Notes from the OCLC Record Use Policy Council discussion

This morning I was at the OCLC Americas Regional Council Meeting just prior to the opening of the ALA Midwinter 2010 meeting. In addition to the prepared talks and remarks, there was a series of breakout sessions the end of the meeting. Ever sense the record use policy kerfuffle got started, I’ve been thinking more about the governenace aspects of OCLC as cooperative, so I attended the session on “The Cooperative’s Shared Values and Social Contract.” It was a very interesting discussion, and for several hours after my mind was spinning with implications of the heartfelt ideas contributed by those at the meeting. In the end, I’m stuck with this line of thinking, starting with a statement then a series of questions:

• I am a librarian at an OCLC member organization.
• Am I, personally, a member of the cooperative?
• If so, and most people seem to speak as if it were so, is there a difference between being a member of the library community and a member of OCLC?
• Can I be a member of one and not the other?
• How are my “member” responsibilities to OCLC the cooperative different from my “membership” responsibilities to the library profession?

The phrase “Social Contract” can be broken down into two pieces: Society and Contract. Taking the latter first, I think of the contract as a shared agreement with and among the membership of the cooperative. So before defining the nature of that contract, we need to define the nature of the society forming the contract. For me, what came out this session was a lack of my own understanding of the definition of the collaborative’s “society” in this instance. What seemed particularly awkward to me was a call-to-action towards the end of the breakout session for OCLC members (as in individuals) to get connected to library schools to tell new professionals about the OCLC shared values and social contract. That struck me, as someone thinking that he maybe sort of outside of the “OCLC membership,” as bordering on indoctrination and coercion. Which is to say that the students signed up to be members of the library profession, not necessarily members of a cooperative, because I’m pretty sure the two are not synonymous.

Perhaps this level of discomfort stems from the fact that I wasn’t around when the cooperative was formed, when it seemed like most people speaking up in the room had been. My introduction to OCLC came right at the conversion from dedicated leased lines to access via the internet. As such, I don’t know of a time before there was a strong cooperative for sharing bibliographic records in real-time.

I also find that I don’t have clarity across another dimension of the “society” term. As OCLC appears to try to chart a new course with governance (the evolution of the Members Council to the Global and Regional Councils) and membership (an increasing focus on international libraries and non-library cultural memory institutions), the membership seems to be struggling with what is the bounds of “society” collective. For me, this can’t help but add to the confusion about who is a member — as an individual — and who isn’t.

So I’m left with these questions. If there are answers — or even other opinions — I’d appreciate pointers to them and discussion in the comments.

Post from: Disruptive Library Technology Jester

What Does It Mean to Be a Member of OCLC?

Most e-mail messages I send are digitally signed using a process called “Pretty Good Privacy“, or PGP. In e-mail applications that don’t understand PGP, this digital signature will show up either as an attachment called “PGP.sig” or as a part of the message starting with “BEGIN PGP SIGNATURE” at the bottom of the e-mail. This file — containing gibberish to the human eye — is used by PGP-aware programs to verify that the message actually came from me. If you are using PGP, I could also sent you a message that only you could read (e.g. “encrypted”). This page gives some background on PGP and why I consider it important.

Background on PGP

My messages are signed with my private key, which is protected by a long and secure password. At any point now or in the future, you could take my message with the PGP.sig file, run it through a PGP program (such as the free GnuPG suite of tools) along with my published public key (see below) and verify that I am the person that sent the message. Normal e-mail doesn’t give you that kind of assurance; the scourge of spam and phishing¹ is a demonstration of the problem that you can’t trust that any average e-mail comes from your relatives or from your bank. By contrast, a verified digitally signed message can give you strong evidence that the message actually came from me.

The Web of Trust

I can also publish the fact that I verified the ownership of someone’s public key. In doing so, I’m telling the world that I have matched a human with a public key and that you can trust it, too. If you believe my verification of that person’s public key, then you too can trust messages signed and encrypted by that key as well. And even if you don’t trust me completely, you might see that three other people have verified the owner of the public key and the combination of the four of us would be enough to convince you of the ownership of that public key — even if you have never met the person. That is the web of trust, and it is popular in software development circles to trust the people submitting patches to code. (For example, the Debian keyring.)

This mechanism for creating trust between individuals is a bottom-up, grass-roots scheme. It relies on one-on-one interactions to extend trust to other individuals. Contrast this with a top-down scheme like SSL that encrypts connections in our web browser². SSL, as commonly implemented, requires a central Certificate Authority to issue a key that is trusted by our web browsers. Our browsers trace the authenticity of the server key to the Certificate Authority key to validate the identify of the web server.

By analogy, another top-down scheme are driver’s licenses. They are issued by a central authority (a state), and as long as you trust the process by which the state issues licenses, you can trust the identity of the person holding the license. A bottom-up analogy might be our human capability of recognizing faces. If I see someone in our office meeting with people that I know work for my organization, I have some confidence that person works for my organization as well.

The web-of-trust gets stronger when more people verify each others public keys. So, needless to say, I’m always looking for people to sign my keys (verify my identity) and am willing to sign the keys of others. Side note: Going to ALA Midwinter in Boston? In addition to exchanging our own signatures, there are a number of people in Boston who are open to key signing exchanges as well. Perhaps a number of us could make an event out of running around the city together…

This Sounds Good. Why Isn’t It Used More?

I don’t know if it will get to a usable form, but I hope so. By using public key signatures on almost all of my messages and by posting this message, I’m hoping to generate awareness and understanding of public key cryptography in general and the PGP technique in particular. At least a little part of my corner of the universe will be aware of it, and given the bottom-up, grass-roots nature of the PGP web-of-trust, perhaps that is a good enough start. If you have questions about PGP and/or run into stumbling blocks try to use it, get in touch with me and I’ll help the best I am able.

My Keys

Peter Murray — Professional

Key ID: 2048D/877838CF

Fingerprint: B021 8300 6844 E459 A18E 83CF 4C7A 6A28 8778 38CF)

Created: 2-Jan-2010; Expires: 5-Feb-2015

Public key as known by keyserver pgp.mit.edu (ASCII-armored version)

Peter Murray — Personal

Key ID: 2048D/4637F6A1

Fingerprint: 5781 5786 7D66 D33B 0F54 D9DE 5820 0CEE 4637 F6A1)

Created: 2-Jan-2010; Expires: 5-Feb-2015

Public key as known by keyserver pgp.mit.edu (ASCII-armored version)

In another DLTJ post, I listed details on how these keys were created.

More Information

Footnotes

1. “Phishing” is a term used to describe techniques used by scammers to try to convince you to give up passwords or other personal information.
2. SSL is an earlier form of the standard now called Transport Layer Security, or TLS.

Post from: Disruptive Library Technology Jester

Why I Digitally Sign My E-Mail

It is the start of a new year¹, and it seems like a good time to update my public encryption key. My previous one — created in 2004 — is both a little weaker, cryptographically speaking, than the ones newly created (1024-bit versus 2048-bit) and also an uncomfortable mixing of my professional and personal lives. For my previous key, I attached all of my professional and personal user ids (e.g. e-mail addresses) to the same key. This time I decided to split my work-related user ids from my other ones. My reasoning for the split is that I might be compelled by my employer to turn over my private key to decrypt messages and files sent in the course of my work. If my personal user ids are also attached to that private key, my employer (and who ever else got ahold of that key), would be able to decrypt my personal messages and files as well. That is not necessarily a good thing. So my solution was to create two keys and cross-sign them. I’ve outlined the process below.

These keys are part of a computer standard and software algorithm called “Pretty Good Privacy“, or PGP. If you are interested in more of a background about PGP, see a companion post on why I digitally sign my e-mail.

The Two New Keys

Peter Murray — Professional

Key ID: 2048D/877838CF

Fingerprint: B021 8300 6844 E459 A18E 83CF 4C7A 6A28 8778 38CF)

Created: 2-Jan-2010; Expires: 5-Feb-2015

Public keys as known by keyserver pgp.mit.edu (ASCII-armored version)

Peter Murray — Personal

Key ID: 2048D/4637F6A1

Fingerprint: 5781 5786 7D66 D33B 0F54 D9DE 5820 0CEE 4637 F6A1)

Created: 2-Jan-2010; Expires: 5-Feb-2015

Public key as known by keyserver pgp.mit.edu (ASCII-armored version)

The key id is a short identifier for the PGP key, and it is broken up into two parts separated by a slash. The first part — “2048D” — says that this is a 2048-bit DSA signing key. The second part — “877838CF” — is the last eight hexadecimal digits of the key fingerprint. Taken together, these two pieces of the key id almost assuredly identify the key. It is a short form, though — the full identifier is the key fingerprint: 40 hexadecimal digits. Embedded in the key are creation and expiration dates. The expiration date can be changed later, and can be eliminated, too (e.g. set to not expire). There doesn’t seem to be much in the way of guidance on how long to set the expiration date, but five years out seems to be a good round number.

Key Creation / Cross-signing Process

Generating the New Key

$ gpg –gen-key
gpg (GnuPG) 2.0.13; Copyright (C) 2009 Free Software Foundation, Inc.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Please select what kind of key you want:
   (1) RSA and RSA (default)
   (2) DSA and Elgamal
   (3) DSA (sign only)
   (4) RSA (sign only)
Your selection? 2
DSA keys may be between 1024 and 3072 bits long.
What keysize do you want? (2048) 2048
Requested keysize is 2048 bits
Please specify how long the key should be valid.
         0 = key does not expire
      <n>  = key expires in n days
      <n>w = key expires in n weeks
      <n>m = key expires in n months
      <n>y = key expires in n years
Key is valid for? (0) 62m
Key expires at Thu Feb  5 13:42:53 2015 EST
Is this correct? (y/N) y

GnuPG needs to construct a user ID to identify your key.

Real name: Peter E. Murray
Email address: peter@OhioLINK.edu
Comment: Professional — supercedes 0×27cf2072
You selected this USER-ID:
    ”Peter E. Murray (Professional — supercedes 0×27cf2072) <peter@OhioLINK.edu>”

Change (N)ame, (C)omment, (E)mail or (O)kay/(Q)uit? o
You need a Passphrase to protect your secret key.

We need to generate a lot of random bytes. It is a good idea to perform
some other action (type on the keyboard, move the mouse, utilize the
disks) during the prime generation; this gives the random number
generator a better chance to gain enough entropy.
gpg: WARNING: some OpenPGP programs can’t handle a DSA key with this digest size
We need to generate a lot of random bytes. It is a good idea to perform
some other action (type on the keyboard, move the mouse, utilize the
disks) during the prime generation; this gives the random number
generator a better chance to gain enough entropy.
gpg: key 877838CF marked as ultimately trusted
public and secret key created and signed.

gpg: checking the trustdb
gpg: 3 marginal(s) needed, 1 complete(s) needed, PGP trust model
gpg: depth: 0  valid:   2  signed:  10  trust: 0-, 0q, 0n, 0m, 0f, 2u
gpg: depth: 1  valid:  10  signed:   2  trust: 0-, 1q, 0n, 4m, 5f, 0u
gpg: depth: 2  valid:   1  signed:   0  trust: 0-, 1q, 0n, 0m, 0f, 0u
gpg: next trustdb check due at 2011-03-13
pub   2048D/877838CF 2010-01-02 [expires: 2015-02-05]
      Key fingerprint = B021 8300 6844 E459 A18E  83CF 4C7A 6A28 8778 38CF
uid                  Peter E. Murray (Professional — supercedes 0×27cf2072) <peter@OhioLINK.edu>
sub   2048g/96854A46 2010-01-02 [expires: 2015-02-05]

Add Other Elements to the New Key

$ gpg –edit-key 877838cf
gpg (GnuPG) 2.0.13; Copyright (C) 2009 Free Software Foundation, Inc.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Secret key is available.

pub  2048D/877838CF  created: 2010-01-02  expires: 2015-02-05  usage: SC  
                     trust: ultimate      validity: ultimate
sub  2048g/96854A46  created: 2010-01-02  expires: 2015-02-05  usage: E   
[ultimate] (1). Peter E. Murray (Professional — supercedes 0×27cf2072) <peter@OhioLINK.edu>

Command> adduid
Real name: Peter E. Murray
Email address: peter.murray@wright.edu
Comment:
You selected this USER-ID:
    ”Peter E. Murray <peter.murray@wright.edu>”

Change (N)ame, (C)omment, (E)mail or (O)kay/(Q)uit? o

pub  2048D/877838CF  created: 2010-01-02  expires: 2015-02-05  usage: SC  
                     trust: ultimate      validity: ultimate
sub  2048g/96854A46  created: 2010-01-02  expires: 2015-02-05  usage: E   
[ultimate] (1)  Peter E. Murray (Professional — supercedes 0×27cf2072) <peter@OhioLINK.edu>
[ unknown] (2). Peter E. Murray <peter.murray@wright.edu>

Command> addphoto

Pick an image to use for your photo ID.  The image must be a JPEG file.
Remember that the image is stored within your public key.  If you use a
very large picture, your key will become very large as well!
Keeping the image close to 240×288 is a good size to use.

Enter JPEG filename for photo ID: ~/pmurray.jpg
Is this photo correct (y/N/q)? y

pub  2048D/877838CF  created: 2010-01-02  expires: 2015-02-05  usage: SC  
                     trust: ultimate      validity: ultimate
sub  2048g/96854A46  created: 2010-01-02  expires: 2015-02-05  usage: E   
[ultimate] (1)  Peter E. Murray (Professional — supercedes 0×27cf2072) <peter@OhioLINK.edu>
[ unknown] (2). Peter E. Murray <peter.murray@wright.edu>
[ unknown] (3)  [jpeg image of size 4916]

Command> 1

pub  2048D/877838CF  created: 2010-01-02  expires: 2015-02-05  usage: SC  
                     trust: ultimate      validity: ultimate
sub  2048g/96854A46  created: 2010-01-02  expires: 2015-02-05  usage: E   
[ultimate] (1)* Peter E. Murray (Professional — supercedes 0×27cf2072) <peter@OhioLINK.edu>
[ unknown] (2). Peter E. Murray <peter.murray@wright.edu>
[ unknown] (3)  [jpeg image of size 4916]

Command> primary

pub  2048D/877838CF  created: 2010-01-02  expires: 2015-02-05  usage: SC  
                     trust: ultimate      validity: ultimate
sub  2048g/96854A46  created: 2010-01-02  expires: 2015-02-05  usage: E   
[ultimate] (1)* Peter E. Murray (Professional — supercedes 0×27cf2072) <peter@OhioLINK.edu>
[ unknown] (2)  Peter E. Murray <peter.murray@wright.edu>
[ unknown] (3)  [jpeg image of size 4916]

Command> save

Generate a Revocation Certificate

$ gpg –output 0×877838cf-revoke.asc –gen-revoke 0×877838cf

sec  2048D/877838CF 2010-01-02 Peter E. Murray (Professional — supercedes 0×27cf2072) <peter@OhioLINK.edu>

Create a revocation certificate for this key? (y/N) yes
Please select the reason for the revocation:
  0 = No reason specified
  1 = Key has been compromised
  2 = Key is superseded
  3 = Key is no longer used
  Q = Cancel
(Probably you want to select 1 here)
Your decision? 1
Enter an optional description; end it with an empty line:
> Revocation cert created at the time of key generation.
>
Reason for revocation: Key has been compromised
Revocation cert created at the time of key generation.
Is this okay? (y/N) y

You need a passphrase to unlock the secret key for
user: “Peter E. Murray (Professional — supercedes 0×27cf2072) <peter@OhioLINK.edu>”
2048-bit DSA key, ID 877838CF, created 2010-01-02

ASCII armored output forced.
Revocation certificate created.

Please move it to a medium which you can hide away; if Mallory gets
access to this certificate he can use it to make your key unusable.
It is smart to print this certificate and store it away, just in case
your media become unreadable.  But have some caution:  The print system of
your machine might store the data and make it available to others!

Cross-signing Keys

The command is “tnrsign”, which stands for a trusted, no-revokable signature. The “trusted” part is a little technical, but it basically means that the signed key can trust the signatures of the signing key. In a practical sense, it means that in my local web-of-trust database, I will continue to trust the keys of people I’ve signed with my old key. (The GnuPG-Users list has a more in-depth discussion of the meaning of trusted signatures.) The “non-revokable” part means, of course, that this signature — this “validation” if you will — cannot be reversed at a later time. In this context, that makes since because I control both ends — the signed key and the signing key.

$ gpg –local-user 0×27cf2072 –edit-key 877838cf tnrsign
gpg (GnuPG) 2.0.13; Copyright (C) 2009 Free Software Foundation, Inc.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Secret key is available.

gpg: checking the trustdb
gpg: 3 marginal(s) needed, 1 complete(s) needed, PGP trust model
gpg: depth: 0  valid:   3  signed:  10  trust: 0-, 0q, 0n, 0m, 0f, 3u
gpg: depth: 1  valid:  10  signed:   2  trust: 0-, 1q, 0n, 4m, 5f, 0u
gpg: depth: 2  valid:   1  signed:   0  trust: 0-, 1q, 0n, 0m, 0f, 0u
gpg: next trustdb check due at 2011-03-13
pub  2048D/877838CF  created: 2010-01-02  expires: 2015-02-05  usage: SC  
                     trust: ultimate      validity: ultimate
sub  2048g/96854A46  created: 2010-01-02  expires: 2015-02-05  usage: E   
[ultimate] (1). Peter E. Murray (Professional — supercedes 0×27cf2072) <peter@OhioLINK.edu>
[ultimate] (2)  Peter E. Murray <peter.murray@wright.edu>
[ultimate] (3)  [jpeg image of size 4916]

Really sign all user IDs? (y/N) yes

pub  2048D/877838CF  created: 2010-01-02  expires: 2015-02-05  usage: SC  
                     trust: ultimate      validity: ultimate
Primary key fingerprint: B021 8300 6844 E459 A18E  83CF 4C7A 6A28 8778 38CF

     Peter E. Murray (Professional — supercedes 0×27cf2072) <peter@OhioLINK.edu>
     Peter E. Murray <peter.murray@wright.edu>
     [jpeg image of size 4916]

This key is due to expire on 2015-02-05.
Please decide how far you trust this user to correctly verify other users’ keys
(by looking at passports, checking fingerprints from different sources, etc.)

  1 = I trust marginally
  2 = I trust fully

Your selection? 2

Please enter the depth of this trust signature.
A depth greater than 1 allows the key you are signing to make
trust signatures on your behalf.

Your selection? 2

Please enter a domain to restrict this signature, or enter for none.

Your selection?

Are you sure that you want to sign this key with your
key “Peter E. Murray <peter@pandc.org>” (27CF2072)

The signature will be marked as non-revocable.

Really sign? (y/N) y

You need a passphrase to unlock the secret key for
user: “Peter E. Murray <peter@pandc.org>”
1024-bit DSA key, ID 27CF2072, created 2004-09-15

Command> save

This command is repeated for the other five combinations of old-new and new-new keys.

Create ASCII-Armored Exports of the New Public Keys

$ gpg –armor –output 877838cf.asc –export 877838cf

Footnotes

1. Some have even said it is the start of a new decade, but of course that isn’t true. We won’t start a new decade until 2011, just like we didn’t actually start a new millennium until 2001.

Post from: Disruptive Library Technology Jester

A New Year, a New PGP Key

Okay, I know this is starting to seem like an obsession, but I can’t figure out why someone(s) would be constructing tweets that consist of my blog post headlines and links back to my postings. I’m wondering how wide spread this problem is, so I constructed a list of URLs to blog posts based on the Planet Code4Lib Atom feed and pointed them to the Ubervu service. Ubervu has a view into the Twitter firehose, and constructs reports of Twitter mentions of URLs. For instance, I can see all of the odd headline tweets for my previous postings through this service. I can then easily scan through the list for other people that seem to be affected by this strange phenomenon.

Eric Schnell has a great summary of these posts and related comments called Is a Twitterfarm Pranking the Jester? in his blog The Medium is the Message. Thank you, Eric.

Here are the results. In all cases except for one, the ‘twitterfeed’ service was used as the bridge between some feed of blog postings into individual tweets.

• Ubervu service view of scienceblogs.com/bookoftrogool/2009/12/making_author_authority_easier.php
• Ubervu service view of elibtronic.ca/content/20091229/anatomy-pown-pownd-part-2
• Ubervu service view of dltj.org/article/twitter-spam/ [Me -- 7 questionable tweets, described in previous post]
• Ubervu service view of people.oregonstate.edu/~reeset/blog/archives/805
• Ubervu service view of peterbrantley.com/reality-dreams-for-libraries-213
• Ubervu service view of commonplace.net/2009/12/old-library-new-library/
• Ubervu service view of orweblog.oclc.org/archives/002040.html
• Ubervu service view of catalogablog.blogspot.com/2009/12/marbi-meeting-minutes.html
• Ubervu service view of people.oregonstate.edu/~reeset/blog/archives/804
• Ubervu service view of www.lisnews.org/librarian_h_o_p_e_hackers_planet_earth_conference
• Ubervu service view of orweblog.oclc.org/archives/001392.html
• Ubervu service view of scienceblogs.com/bookoftrogool/2009/12/top-down_or_bottom-up.php
• Ubervu service view of dltj.org/article/alamw10-schedule/ [Me again -- 5 questionable tweets, described in previous post]
• Ubervu service view of infomotions.com/blog/2009/12/good-and-best-open-source-software/ [2 questionable tweets: Twitter ID 'audio_college' (3,240 followers) and ID 'DTcomputers' (10,204 followers)]
• Ubervu service view of catalogablog.blogspot.com/2009/12/isbd-area-0-in-rusian.html [1 questionable tweet: Twitter ID 'rem_simanovski' (658 followers)
• Ubervu service view of go-to-hellman.blogspot.com/2009/12/case-against-using-spoofed-e-books-to.html [1 questionable tweet: Twitter ID 'ispicey' (8,177 followers)
• Ubervu service view of orweblog.oclc.org/archives/002039.html
• Ubervu service view of orweblog.oclc.org/archives/002038.html
• Ubervu service view of orweblog.oclc.org/archives/002037.html [4 questionable tweets: Twitter ID 'peterpains' (1,017 followers); 'LostInDaSources' (1,057 followers, other posted links in twitter stream to spamy web pages); 'JackOOler' (914 followers); 'FreePsyche' (123 followers, is adding text to blog post titles in tweet)]
• Ubervu service view of john.mignault.net/blog/2009/12/26/coffee-exchange/
• Ubervu service view of bibwild.wordpress.com/2009/12/24/issn-search-field-in-solr/
• Ubervu service view of people.oregonstate.edu/~reeset/blog/archives/801
• Ubervu service view of www.libraryjournal.com/blog/1090000309/post/1950051395.html?nid=3565
• Ubervu service view of hublog.hubmed.org/archives/001891.html
• Ubervu service view of catalogablog.blogspot.com/2009/12/cookery.html
• Ubervu service view of catalogablog.blogspot.com/2009/12/character-sets.html
• Ubervu service view of go-to-hellman.blogspot.com/2009/12/holiday-product-management-and.html [1 questionable tweet: Twitter ID 'holiday_gifts' (3,237 followers)]
• Ubervu service view of community.oclc.org/hecticpace/archive/2009/12/jingle-books.html
• Ubervu service view of ptsefton.com/2009/12/23/bye-bye-word-2007-custom-xml.htm
• Ubervu service view of scienceblogs.com/bookoftrogool/2009/12/tidbits_22_december_2009.php
• Ubervu service view of inkdroid.org/journal/2009/12/22/hacking-oreilly-rdfa/ 1[ questionable tweet: Twitter ID 'soslab' (122 followers, also reposted a DLTJ post)]
• Ubervu service view of www.eff.org/deeplinks/2009/12/e-book-privacy
• Ubervu service view of litablog.org/2009/12/last-standards-announcements-of-2009/
• Ubervu service view of catalogablog.blogspot.com/2009/12/anatomy-of-catalog-record.html
• Ubervu service view of efoundations.typepad.com/efoundations/2009/12/online-learning-in-virtual-environments-final-report.html
• Ubervu service view of blog.iandavis.com/2009/12/new-blog-url-blog-iandavis-com
• Ubervu service view of catalogablog.blogspot.com/2009/12/cuttering-at-national-library-of.html [1 questionable tweet: Twitter ID 'jadep2008' (account now suspended by Twitter)]
• Ubervu service view of community.oclc.org/hecticpace/archive/2009/12/hectic-shame.html
• Ubervu service view of miskatonic.org/2009/12/21/my-code4lib-2010-t-shirt
• Ubervu service view of miskatonic.org/2009/12/21/dchud-and-nunanishi
• Ubervu service view of mblog.lib.umich.edu/blt/archives/2009/12/further_tweaks.html [1 questionable tweet: Twitter ID 'FastestFood' (124 followers, profile URL leads to "get rich quick" site)]
• Ubervu service view of blog.threepress.org/2009/12/21/nook-1-1-0-firmware-update-report/
• Ubervu service view of maisonbisson.com/blog/post/14198/apple-netbook-newton-emate-300/
• Ubervu service view of go-to-hellman.blogspot.com/2009/12/copyright-enforcement-for-ebooks.html [1 questionable tweet: Twitter ID 'OnlineTVNews' (160 followers, using "RSS2Twitter" rather than twitterfeed, profile URL points to a commercial TV-over-IP service)]
• Ubervu service view of efoundations.typepad.com/efoundations/2009/12/scanning-horizons-for-the-semantic-web-in-higher-education.html
• Ubervu service view of code4lib.org/node/346
• Ubervu service view of orweblog.oclc.org/archives/002036.html
• Ubervu service view of orweblog.oclc.org/archives/002035.html [1 questionable tweet: Twitter ID 'workfanatic' (878 followers, tweets include part of post, may be a legitimate consulting service)]
• Ubervu service view of john.mignault.net/blog/2009/12/20/calibre-quickstart-for-kindle/
• Ubervu service view of www.frbr.org/2009/12/20/last-week-in-frbr-11 [2 questionable tweets: Twitter ID 'soslab' (122 followers, also seen with other Code4Lib planet posts); 'CTSeven' (3,968 followers, profile URL seems to point to a legitimate business)]
• Ubervu service view of www.libraryjournal.com/blog/1090000309/post/300051430.html?nid=3565
• Ubervu service view of www.parser.ca/z678/2009/12/18/this-is-what-im-talking-about-evergreen-ils/ [3 questionable tweets, all with the same profile URL that seems to point to a legitimate business: Twitter ID 'GlowPaint' (1,559 followers, prepends fixed text string to tweets); 'BlackLightInfo' (943 followers); 'FutureOfGlow' (5,090, prepends fixed text string to tweets)]
• Ubervu service view of scienceblogs.com/bookoftrogool/2009/12/authority_control_then_and_now.php
• Ubervu service view of mblog.lib.umich.edu/blt/archives/2009/12/hathitrust_reac.html
• Ubervu service view of futurearchives.blogspot.com/2009/12/virtually-jodconverter-ii.html [1 questionable tweet: Twitter ID 'archivesopen' (701 followers, profile URI points to a Blogspot blog, seems to be legitimate)]
• Ubervu service view of litablog.org/2009/12/lita-happy-hour-mw2010/ [1 questionable tweet: Twitter ID 'library_breath' (253 followers, profile URL points to Japanese site)]
• Ubervu service view of catalogablog.blogspot.com/2009/12/xforms4lib.html
• Ubervu service view of www.libraryjournal.com/blog/1090000309/post/1840051384.html?nid=3565
• Ubervu service view of vphill.com/journal/?p=2740
• Ubervu service view of digitalcuration.blogspot.com/2009/12/more-activity-on-semantic-publishing.html
• Ubervu service view of www.librarywebchic.net/wordpress/2009/12/17/learning-and-loving-jquery-for-the-most-part/ [1 questionable tweet: Twitter ID 'clearvisage' (1,405 followers, profile URL points to a "skin rejuvenation" site)]
• Ubervu service view of futurearchives.blogspot.com/2009/12/virtually-jodconverter.html [1 questionable tweet: Twitter ID 'careersoft' (2,648 followers, profile URL points to a site without a DNS entry)]
• Ubervu service view of futurearchives.blogspot.com/2009/12/t.html [2 questionable tweets from the same account: Twitter ID 'archivesopen' (701 followers, profile URI points to a Blogspot blog, seems to be legitimate)]

Interestingly, in one case — inkdroid.org/journal/2009/12/22/hacking-oreilly-rdfa/ — ‘twitterfeed’ seem to be legitimately used by Eqentia for a Twitter account called ’semanticnews’. The bio on the twitter account says: “Tracking what’s new in the Semantic Web space. 2,500+ articles indexed via Eqentia’s semantic platform. Sign-up and experience Semantic-powered News”. Ubervu also shows that the ’semanticnews’ tweet was the start of a Twitter thread of three other tweets on the same topic.

Analysis

I went poking in my server’s access logs searching for occurrences of ‘twitterfeed’ and came back with a surprise: where I expected to see ‘twitterfeed’ in the User-Agent string, I actually found it more as a Google Analytics parameter on URL requests in these two forms:

• ?utm_source=twitterfeed&utm_medium=twitter
• ?utm_source=GAlert&utm_medium=twitterfeed&utm_campaign=CDT_RSS&utm_term=TechNews

At this point, I’m not sure what is introducing those parameters. I can’t find documentation for it in the Google Analytics help system, but I suspect it might be coming as a part of Feedburner. I’m pretty much a newbie when it comes to Google Analytics, so if anyone has any insights, I’d appreciate it.

There are two cases where ‘twitterfeed’ is being used as part of a user agent string (or "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-GB; rv:1.8.1.3) Gecko/20070309 Firefox/2.0.0.3 twitterfeed" more specifically). I’m going to set up a honeypot for twitterfeed using mod_rewrite conditions on my server:

## Attempt to block twitterfeed
RewriteCond %{USER_AGENT} "twitterfeed"
RewriteRule feed.* /atom-feed-for-twitterfeed.xml [R=302,L]

The “atom-feed-for-twitterfeed.xml” file consists of:

< ?xml version="1.0" encoding="utf-8" standalone="yes" ?>
<feed xmlns="http://www.w3.org/2005/Atom">
 
	<title>DLTJ Twitter Honeypot</title>
	<link rel="alternate" type="text/html" href="http://dltj.org"/>
	<id>http://dltj.org</id>
	<updated>2009-12-30T17:47:10+00:00</updated>
	<generator uri="http://dltj.org/about/">An annoyed jester</generator>
 
	<entry>
		<title>Twitter and Twitterfeed honeypot</title>
		<link rel="alternate" type="text/html"
			href="http://dltj.org/article/questionable-twitter-posts/"/>
		<id>http://dltj.org/article/questionable-twitter-posts/</id>
		<updated>2009-12-30T17:28:07+00:00</updated>
		<content type="html">&lt;p&gt;This is a honeypot to try to catch Twitterfeed when it injects postings into Twitter.  For more information on why I'm trying this, see &lt;a href="http://dltj.org/article/questionable-twitter-posts/"&gt;this blog post on &lt;acronym title="Disruptive Library Technology Jester"&gt;&lt;i&gt;DLTJ&lt;/i&gt;&lt;/acronym&gt;&lt;/a&gt;.&lt;/p&gt;</content>
		<author>
			<name>Murray, Peter</name>
			<uri>http://dltj.org/about</uri>
		</author>
	</entry>
</feed>

Yeah — I know I’m breaking the rules by giving different content for the same URI. But remember, this is just a honeypot.

With this, I’m going to see if my honeypot entry shows up in one of these Twitterfeed-injected posts. Am I showing signs of being obsessed with this? Yep, no doubt. But I really want to know how and where my content is being used. This itch definitely needs to be scratched.

Post from: Disruptive Library Technology Jester

On Being Fodder for Questionable Twitter Posts

The following may not be news to those who regularly hang out in Twitter-land, but the extent of the problem recently became clear to me: there is a bunch of spam in Twitter. More specifically, there appear to be robots that do nothing but scan the web for keywords and create tweets with links back to them. There appear to be some that value this service (judging by the number of followers of these Twitter users), but for me it just adds to the general clutter I find in Twitter.

So — here is the situation. Yesterday I posted a blog message that has my upcoming ALA Midwinter meeting plans. I’ve got a WordPress plugin that injects an announcement of that post into my Twitter stream. Since I like my blog to be the definitive source of discussions surrounding my blog posts, I also run another plug-in (from the Backtype service) that takes commentary found in other social media sites and adds them as comments to my blog posting. I’ve set the latter plug-in to add such comments to my “pending” queue rather than posting them automatically.

When I looked at my pending comment queue this morning, I saw that Backtype found not only my own tweet of the post¹, but also five others from “people” I haven’t encountered before. (No links here because I don’t want to offer any Google juice if there is something nefarious going on.)

A couple of things to note:

• In all cases, the Twitter IDs seem to be unlike other spammers — ones that I typically associate with spammers are names with a string of numbers. These look like real names.
• Three of the five accounts have over 1,000 followers — usually the mark of someone legitimate. Heck…that is more than I have by far!
• Three accounts (TechnoTrendz, ddaville, and FrankyConnelly) also add an excerpt of text from deep inside the post: “Next planned event is the discussion mee” Two of these three use the same bit.ly short link.
• The original post did not use a third-party URL shortener.² These five posts contain 3 unique bit.ly short links, with three of the five using the same short link.

All told, this looks suspicious. It is also the sort of thing that leads me to use third-party tools to distill Twitter content into something more manageable and less spam-y. Have others noticed the same thing? Do you have any coping strategies for dealing with the Twitter stream?

Evening Update

So the question would be — for what purpose? To as fodder to mask truly spamy tweets? Because the account owner thinks their followers might all be interested in what I’m saying? What I do know is that this practice — at least for my blog posts — has increased dramatically in the past few weeks. I don’t think this was happening earlier this month…

Footnotes

1. Oddly, I didn’t get a tweet from InfoPeep — the reposting service based on the Code4Lib Planet.
2. I’m happy I have an inherently short URL to start with, so am using yet another WordPress plugin to internally direct users from short URLs to canonical ones.

Post from: Disruptive Library Technology Jester

Why I Need Twitter Distillation Tools